tensorflow: memory allocation for a 'for' cycle
<p>I am trying to use TensorFlow for calculating minimum Euclidean distance between each column in the matrix and all other columns (excluding itself):</p>

<pre><code>with graph.as_default():
  ...
  def get_diversity(matrix):
       num_rows = matrix.get_shape()[0].value
       num_cols = matrix.get_shape()[1].value
       identity = tf.ones([1, num_cols], dtype=tf.float32)
       diversity = 0

       for i in range(num_cols):
           col = tf.reshape(matrix[:, i], [num_rows, 1])
           col_extended_to_matrix = tf.matmul(neuron_matrix, identity)
           difference_matrix = (col_extended_to_matrix - matrix) ** 2
           sum_vector = tf.reduce_sum(difference_matrix, 0)
           mask = tf.greater(sum_vector, 0)
           non_zero_vector = tf.select(mask, sum_vector, tf.ones([num_cols], dtype=tf.float32) * 9e99)
           min_diversity = tf.reduce_min(non_zero_vector)
           diversity += min_diversity

       return diversity / num_cols
  ...

  diversity = get_diversity(matrix1)

  ...
</code></pre>

<p>When I call <code>get_diversity()</code> once per 1000 iterations (on the scale of 300k) it works just fine. But when I try to call it at every iteration the interpreter returns:</p>

<pre><code>W tensorflow/core/common_runtime/bfc_allocator.cc:271] Ran out of memory trying to allocate 2.99MiB.  See logs for memory state.
</code></pre>

<p>I was thinking that was because TF creates a new set of variables each time  <code>get_diversity()</code> is called. I tried this:</p>

<pre><code>def get_diversity(matrix, scope):
    scope.reuse_variables()
...
with tf.variable_scope("diversity") as scope:
    diversity = get_diversity(matrix1, scope)
</code></pre>

<p>But it did not fix the problem. </p>

<p>How can I fix this allocation issue and use <code>get_diversity()</code> with large number of iterations?</p>