GPU + CPU Tensorflow Training
<p><strong>Setup</strong></p>

<hr>

<p>I have a network, one whose parameter is a <strong>large-embedding matrix (3Million X 300 sized)</strong>, say embed_mat. </p>

<p>During training, for each mini-batch, I only update a <strong>small subset</strong> of the vectors from embed_mat (max 15000 vectors) which are chosen using the <strong>embedding_lookup op</strong>. I am using the <strong>Adam optimizer</strong> to train my model. </p>

<p>As I cannot store this embed_mat in the GPU, due to its size, I define it under CPU (say /cpu:0) device, but the rest of the parameters of the model, the optimizer etc. are defined under a GPU (say, gpu:/0) device. </p>

<p><strong>Questions</strong></p>

<hr>

<ol>
<li><p>I see that my GPU usage is very minimal (200 MB), which suggests all my training is happening on the CPU. What I expected was that the result of the embedding_lookup is copied to the GPU and all my training happens there. Am I doing something wrong.</p></li>
<li><p>The training time is very largely affected by the size (num_vectors) of the embedding matrix which doesn't seem correct to me. In any mini-batch, I only update my network parameters and the vectors I looked up (~15000), so the training time should, if at all, grow sub-linearly with the size of the embedding matrix.</p></li>
<li><p>Is there a way to automatically and seamlessly split up my embed_mat to multiple GPUs for faster training?</p></li>
<li><p>I suspect the Adam Optimizer for this. Looks like because the embed_mat is on the CPU, all training is happening on the CPU. Is this correct?</p></li>
</ol>