Restoring checkpoint in distributed tensorflow
<p>Using a setup similar to <a href="https://github.com/tensorflow/models/tree/master/inception" rel="nofollow">https://github.com/tensorflow/models/tree/master/inception</a>, the chief worker automatically saves a checkpoint file periodically on the node this process is running on. I'm running two ps on two different nodes. Two workers are also running on the two nodes each, with one out of 4 workers being the chief.</p>

<p>When restarting training without any modification, the Supervisor automatically tries to restore the last checkpoint file, but ends up giving an error that it could not find the ckpt on the second node (the node other than the chief worker), because the chief never saved the ckpt on the second node.</p>

<pre><code>W tensorflow/core/framework/op_kernel.cc:936] Not found: Unsuccessful TensorSliceReader constructor: Failed to find any matching files for /home/muneebs/tf_train/model.ckpt-275
</code></pre>

<p>If I copy the ckpt directory to the second node, it restores fine. Is it a bug? Should the saver be initialized as sharded=True? If so, is that the only way, and we can't have the ckpt as a single file in case the number of nodes change later on?</p>