Tesla K80 memory usage with TensorFlow
<p>I've run a sample code of convolutional neural networks for TensorFlow with Tesla k80 GPU's.</p>

<p><a href="https://www.tensorflow.org/versions/r0.9/tutorials/deep_cnn/index.html#launching-and-training-the-model-on-multiple-gpu-cards" rel="nofollow">https://www.tensorflow.org/versions/r0.9/tutorials/deep_cnn/index.html#launching-and-training-the-model-on-multiple-gpu-cards</a></p>

<pre><code>$ python cifar10_multi_gpu_train.py --batch_size 64 --num_gpus 2

Then I've checked a memory usage of GPU's. The result is below.

$ nvidia-smi 
Mon Oct 31 16:34:15 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.93     Driver Version: 352.93         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K420         Off  | 0000:01:00.0      On |                  N/A |
| 25%   45C    P0    N/A /  N/A |    264MiB /  1023MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           Off  | 0000:04:00.0     Off |                    0 |
| N/A   41C    P0    66W / 149W |  11022MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla K80           Off  | 0000:05:00.0     Off |                    0 |
| N/A   37C    P0    67W / 149W |    130MiB / 11519MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      8757    C   python                                           8MiB |
|    0     13398    G   /usr/bin/Xorg                                   92MiB |
|    0     14355    G   /usr/bin/gnome-shell                           146MiB |
|    1      8757    C   python                                       10956MiB |
|    2      8757    C   python                                          64MiB |
+-----------------------------------------------------------------------------+
</code></pre>

<p>Memory usage of the "GPU 1" was 11Gb which is ~95% of maximum of the 12Gb. But the memory usage of the "GPU 2" only 64Mb. 
Could you tell me how to use both GPU's memory on the Tesla K80 with TensorFlow ?    </p>