Analysis of NaN Divergence in Training Monocular Depth Estimation Model
Bum Jun Kim Hyeonah Jang Sang Woo Kim
The latest advances in deep learning have facilitated the development of
highly accurate monocular depth estimation models. However, when training a
monocular depth estimation network, practitioners and researchers have observed
not a number (NaN) loss, which disrupts gradient descent optimization. Although
several practitioners have reported the stochastic and mysterious occurrence of
NaN loss that bothers training, its root cause is not discussed in the
literature. This study conducted an in-depth analysis of NaN loss during
training a monocular depth estimation network and identified three types of
vulnerabilities that cause NaN loss: 1) the use of square root loss, which
leads to an unstable gradient; 2) the log-sigmoid function, which exhibits
numerical stability issues; and 3) certain variance implementations, which
yield incorrect computations. Furthermore, for each vulnerability, the
occurrence of NaN loss was demonstrated and practical guidelines to prevent NaN
loss were presented. Experiments showed that both optimization stability and
performance on monocular depth estimation could be improved by following our
guidelines.