Tensorflow: Why Doesn't Out of Graph Cost Calculation Work
<p>I have a standard experiment loop that looks like this:</p>

<pre class="lang-py prettyprint-override"><code>cross_entropy_target = tf.reduce_mean(tf.reduce_mean(tf.square(target_pred - target)))
cost = cross_entropy_target
opt_target = tf.train.AdamOptimizer(learning_rate=0.00001).minimize(cost)
for epoch in range(num_epochs):
    for mini_batch in range(num_samples / batch_size):
        mb_train_x, mb_train_target = get_mini_batch_stuffs()
        sess.run(opt_target, feed_dict={x: mb_train_x, target: mb_train_target})
</code></pre>

<p>This runs and converges to a good prediction loss.  Now, same code with a slight modification: </p>

<pre class="lang-py prettyprint-override"><code>cross_entropy_target = tf.reduce_mean(tf.reduce_mean(tf.square(target_pred - target)))
cross_entropy_target_variable = tf.Variable(0.0)
cost = cross_entropy_target_variable
opt_target = tf.train.AdamOptimizer(learning_rate=0.00001).minimize(cost)
for epoch in range(num_epochs):
    for mini_batch in range(num_samples / batch_size):
        mb_train_x, mb_train_target = get_mini_batch_stuffs()
        new_target_cost = sess.run(cross_entropy_target, feed_dict={x: mb_train_x, time: mb_train_time, target: mb_train_target})
        sess.run(tf.assign(cross_entropy_target_variable, new_target_cost))
        sess.run(opt_target, feed_dict={x: mb_train_x, target: mb_train_target})
</code></pre>

<p>Now, instead of the <code>cross_entropy_target</code> being calculated as part of the  <code>opt_target</code> graph, I am pre-calculating it, assigning it to a tensorflow variable, and expecting it to make use of that value.  This doesn't work at all.  The network's outputs never change.  </p>

<p>I would expect these two code snippets to have equivalent outcomes. In both cases a feed forward is used to populate the values of <code>target</code> and <code>target_pred</code>, which is then reduced to the scalar value <code>cross_entropy_target</code>. This scalar value is used to inform the magnitude and direction of the gradient updates on the optimizer's <code>.minimize()</code>.</p>

<p>In this toy example there is no advantage to my calculating the cross_entropy_target "out of graph" and then assigning it to an in-graph <code>tf.Variable</code> for use in the <code>opt_target</code> run.  However, I have a real use case where my cost function is very complex and I have not been able to define it in terms of Tensorflow's existing tensor transforms.  Either way, I'd like to understand why using a <code>tf.Variable</code> for an optimizer's cost is incorrect use.</p>

<p>An interesting oddity that may be a byproduct of the solution to this:
If I set <code>cross_entropy_target_variable = tf.Variable(0.0, trainable=False)</code>, running the <code>opt_target</code> will crash.  It requires that the cost value is modifiable.  Indeed, printing out its value before and after running the <code>opt_target</code> produces different values:</p>

<p><code>cross_entropy_target before = 0.345796853304
cross_entropy_target after = 0.344796866179
</code></p>

<p>Why does running <code>minimize()</code> modify the value of the cost variable?</p>