How to solve nan loss?
<h1>Problem</h1>

<p>I'm running a Deep Neural Network on the MNIST where the loss defined as follow:</p>

<p><code>cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(pred, label))</code></p>

<p>The program seems to run correctly until I get a nan loss in the 10000+ th minibatch. Sometimes, the program runs correctly until it finished. I think <code>tf.nn.softmax_cross_entropy_with_logits</code> is giving me this error.
This is strange, because the code just contains <code>mul</code> and <code>add</code> operations.</p>

<h1>Possible Solution</h1>

<p>Maybe I can use:</p>

<pre><code>if cost == "nan":
  optimizer = an empty optimizer 
else:
  ...
  optimizer = real optimizer
</code></pre>

<p>But I cannot find the type of <code>nan</code>. How can I check a variable is <code>nan</code> or not?</p>

<p>How else can I solve this problem?</p>