Distributing graphs across several machines in Distributed Tensorflow
<p>I am currently working on a project using Distributed Tensorflow. My goal is to run several independent graphs across several different machines.</p>

<p>As an example, I want to do something like this (assume that the server is open on each machine)</p>

<pre><code>import tensorflow as tf
a = tf.constant(3)
b = tf.constant(2)
x = tf.mul(a,b)             # To be run on "grpc://www.example0.com:2222"
y = tf.mul(a,b)             # To be run on "grpc://www.example1.com:2222"
z = tf.mul(a,b)             # To be run on "grpc://www.example2.com:2222"

with tf.Session() as sess:
    sess.run([x,y,z])       # Ops x,y,z are run on different machines in parallel
</code></pre>

<p>My current attempt at this is shown in the following code. However, this code runs the sessions in serial, but I want them to be executed in a parallel distributed manner.</p>

<pre><code>import tensorflow as tf
a = tf.constant(3)
b = tf.constant(2)
x = tf.mul(a,b)             # To be run on "grpc://www.example0.com:2222"
y = tf.mul(a,b)             # To be run on "grpc://www.example1.com:2222"
z = tf.mul(a,b)             # To be run on "grpc://www.example2.com:2222"

with tf.Session("grpc://www.example0.com:2222") as sess:
    sess.run(x)
with tf.Session("grpc://www.example1.com:2222") as sess:
    sess.run(y)
with tf.Session("grpc://www.example2.com:2222") as sess:
    sess.run(z)
</code></pre>

<p>While reading the documentation about Distributed Tensorflow, I found that <code>tf.device</code> allows me to set which CPU or GPU to run Tensorflow Ops on. Is there something similar that allows me to set the <code>session target</code> to specify which machine will run which op? Or is there another way of distributing Tensorflow Ops?</p>