Understanding the conceptual basics of Distributed TensorFlow
<p>Let me describe the cluster setup first :</p>

<ul>
<li>I have two nodes (each with 2 GPUs). I refer to them as Node A and Node B</li>
<li>Each node has its own SSD storage.</li>
<li><a href="http://oar.imag.fr/docs/latest/user/commands/oarsub.html" rel="nofollow noreferrer">OAR</a> is the cluster manager that is used.</li>
</ul>

<p>I have gone through the Distributed TensorFlow documentation but there are some functional basics I could not understand properly and hence this question.</p>

<p>Consider the following situation :</p>

<ul>
<li>I have copied around 600 GB of data on Node A.</li>
<li>I can use OAR to specifically ask for allocation of 4 GPUs across the two nodes.</li>
</ul>

<p>If I want to use Distributed TensorFlow to train a model :</p>

<ol>
<li>How do I specify network addresses to tf.train.ClusterSpec ? What are those network addresses ? In the documentation are names such as localhost:2222 the same names reserved for a particular node with the cluster manager ?</li>
<li>My data is copied to node A. During training will TensorFlow itself be responsible for sending this data as input to the GPU that is on node B ?</li>
<li>Will I need to manually create the TensorFlow Graph for each GPU on each node using tf.device() ?</li>
<li>If I also want to use some additional CPU nodes will I have to have their names beforehand and put them in the code ?</li>
</ol>