<SYSTEM_TASK:>
Given the following text description, write Python code to implement the functionality described below step by step
<END_TASK>
<USER_TASK:>
Description:
 Train Model on Distributed Cluster
IMPORTANT
Step1:  Start Server "Task 0" (localhost
Step2:  Start Server "Task 1" (localhost
Step3:  Define Compute-Heavy TensorFlow Graph
Step4:  Define Shape
Step5:  Assign Devices Manually
All CPU Devices
Note the execution time.
Step6:  CPU and GPU
Note the execution time.
Step7:  All GPU Devices
Note the execution time.
Step8:  Auto-assign Device by TensorFlow (Round-Robin by Default)
Note the execution time.
<ASSISTANT_TASK:>
Python Code:
import tensorflow as tf
cluster = tf.train.ClusterSpec({"local": ["localhost:2222", "localhost:2223"]})
Explanation: Train Model on Distributed Cluster
IMPORTANT:  You Must STOP All Kernels and Terminal Session
The GPU is wedged at this point.  We need to set it free!!
Define ClusterSpec
End of explanation
server0 = tf.train.Server(cluster, job_name="local", task_index=0)
print(server0)
Explanation: Start Server "Task 0" (localhost:2222)
Note:  If you see UnknownError: Could not start gRPC server, then you have already started the server.  Please ignore this.
End of explanation
server1 = tf.train.Server(cluster, job_name="local", task_index=1)
print(server1)
Explanation: Start Server "Task 1" (localhost:2223)
Note:  If you see UnknownError: Could not start gRPC server, then you have already started the server.  Please ignore this.
End of explanation
import tensorflow as tf
n = 2
c1 = tf.Variable([])
c2 = tf.Variable([])
def matpow(M, n):
    if n < 1: 
        return M
    else:
        return tf.matmul(M, matpow(M, n-1))
Explanation: Define Compute-Heavy TensorFlow Graph
End of explanation
shape=[2500, 2500]
Explanation: Define Shape
End of explanation
import datetime
with tf.device("/job:local/task:0/cpu:0"):
    A = tf.random_normal(shape=shape)
    c1 = matpow(A,n)
with tf.device("/job:local/task:1/cpu:0"):
    B = tf.random_normal(shape=shape)
    c2 = matpow(B,n)
with tf.Session("grpc://127.0.0.1:2222") as sess:
    sum = c1 + c2
    start_time = datetime.datetime.now()
    print(sess.run(sum))
    print("Execution time: " 
          + str(datetime.datetime.now() - start_time))
          
Explanation: Assign Devices Manually
All CPU Devices
Note the execution time.
End of explanation
with tf.device("/job:local/task:0/gpu:0"):
    A = tf.random_normal(shape=shape)
    c1 = matpow(A,n)
with tf.device("/job:local/task:1/cpu:0"):
    B = tf.random_normal(shape=shape)
    c2 = matpow(B,n)
with tf.Session("grpc://127.0.0.1:2222") as sess:
    sum = c1 + c2
    start_time = datetime.datetime.now()
    print(sess.run(sum))
    print("Execution time: " 
          + str(datetime.datetime.now() - start_time))
Explanation: CPU and GPU
Note the execution time.
End of explanation
with tf.device("/job:local/task:0/gpu:0"):
    A = tf.random_normal(shape=shape)
    c1 = matpow(A,n)
with tf.device("/job:local/task:1/gpu:0"):
    B = tf.random_normal(shape=shape)
    c2 = matpow(B,n)
with tf.Session("grpc://127.0.0.1:2222") as sess:
    sum = c1 + c2
    start_time = datetime.datetime.now()
    print(sess.run(sum))
    print("Execution time: " 
          + str(datetime.datetime.now() - start_time))
Explanation: All GPU Devices
Note the execution time.
End of explanation
with tf.device(tf.train.replica_device_setter(worker_device="/job:worker/task:0",
                                              cluster=cluster)):
    A = tf.random_normal(shape=shape)
    c1 = matpow(A,n)
with tf.device(tf.train.replica_device_setter(worker_device="/job:worker/task:1",
                                              cluster=cluster)):
    B = tf.random_normal(shape=shape)
    c2 = matpow(B,n)
with tf.Session("grpc://127.0.0.1:2222") as sess:
    sum = c1 + c2
    start_time = datetime.datetime.now()
    print(sess.run(sum))
    print("Multi node computation time: " 
          + str(datetime.datetime.now() - start_time))
Explanation: Auto-assign Device by TensorFlow (Round-Robin by Default)
Note the execution time.
End of explanation
<END_TASK>