What is the advantage of doing a Multi-GPU training in TensorFlow?
<p>In <a href="https://www.tensorflow.org/versions/r0.9/tutorials/deep_cnn/index.html" rel="nofollow">this TensorFlow tutorial</a>, you can use N number of GPUs to distribute N mini-batches (each containing M training samples) to each GPU and calculate the gradients concurrently.</p>

<p>Then you average the gradients collected from N GPUs and update the model parameters.</p>

<p>But this has the same effect as using a single GPU to calculate the gradients of N*M training samples, then updating the parameters.</p>

<p>So the only advantage seems to me is that you can use a larger-sized mini-batch in the same amount of time.</p>

<p>But is the larger-sized mini-batch necessarily better?</p>

<p>I thought you shouldn't use a large-sized mini-batch, in order to make the optimization more robust to saddle points.</p>

<p>If the larger-sized mini-batch is indeed not better, why would you care about Multi-GPU learning, or even Multi-server learning?</p>

<p>(The tutorial above is a synchronous training. If it was asynchronous training, then I can see the merit, since the parameters will be updated without averaging the gradients calculated by each GPU)</p>