How to implement Hogwild! style updates in Tensorflow
<p>I am trying to implement a simple hogwild proof of concept in Tensorflow. Ideally it would use Python threads to perform updates. I have an implementation already done, which follows closely Deep MNIST for experts tutorial from TF website, except with Hogwild. The implementation is available here: <a href="https://github.com/nivwusquorum/a3c/blob/master/hogwild/mnist.py" rel="nofollow">https://github.com/nivwusquorum/a3c/blob/master/hogwild/mnist.py</a></p>

<p>Crucial section of code is here:</p>

<pre><code>def accuracy(session, graphs, data_iter, num_threads, train=False):
    num_total   = 0
    num_correct = 0

    def process_batch(batch_x, batch_y):
        nonlocal num_correct
        nonlocal num_total
        with graphs.lease() as g:
            input_placeholder, output_placeholder, \
            keep_prob_placeholder, train_step_f, num_correct_f, \
            no_op = g
            batch_num_correct, _ = session.run(
                [num_correct_f, train_step_f if train else no_op],
                {
                    input_placeholder:     batch_x,
                    output_placeholder:    batch_y,
                    keep_prob_placeholder: 0.5 if train else 1.0,
                })
            num_correct += batch_num_correct
            num_total   += len(batch_x)

    with BlockOnFullThreadPool(max_workers=num_threads, queue_size=num_threads // 2) as pool:
        for i, (batch_x, batch_y) in enumerate(data_iter):
            pool.submit(process_batch, batch_x, batch_y)
        pool.shutdown(wait=True)

    return float(num_correct) / float(num_total)
</code></pre>

<p>So essentially I create <code>num_threads</code> sets of tensorflow expressions (in my code they are poorly named - graphs), where the variables are reused and optimizer is not using locking. Now the implementation has all the desired characteristics (parallelization speed-up, performance at convergence etc.), but every time I run it it diverges around epoch 30, after achieving optimal performance, which is only the case for <code>num_threads=10</code> and not the case for <code>num_threads=1</code> (training/validation per epoch log for <code>num_threads=10</code> is available here: <a href="http://pastebin.com/J1DqAtz8" rel="nofollow">http://pastebin.com/J1DqAtz8</a>). </p>

<p>Does anybody know what may have cause that? Do I miss something fundamental? For example does every graph compute gradients in separate arrays? </p>