Loss Spike in Training Neural Networks
Zhongwang Zhang Zhi-Qin John Xu
In this work, we study the mechanism underlying loss spikes observed during
neural network training. When the training enters a region, which has a
smaller-loss-as-sharper (SLAS) structure, the training becomes unstable and
loss exponentially increases once it is too sharp, i.e., the rapid ascent of
the loss spike. The training becomes stable when it finds a flat region. The
deviation in the first eigen direction (with maximum eigenvalue of the loss
Hessian ($\lambda_{\mathrm{max}}$) is found to be dominated by low-frequency.
Since low-frequency is captured very fast (frequency principle), the rapid
descent is then observed. Inspired by our analysis of loss spikes, we revisit
the link between $\lambda_{\mathrm{max}}$ flatness and generalization. For real
datasets, low-frequency is often dominant and well-captured by both the
training data and the test data. Then, a solution with good generalization and
a solution with bad generalization can both learn low-frequency well, thus,
they have little difference in the sharpest direction. Therefore, although
$\lambda_{\mathrm{max}}$ can indicate the sharpness of the loss landscape,
deviation in its corresponding eigen direction is not responsible for the
generalization difference. We also find that loss spikes can facilitate
condensation, i.e., input weights evolve towards the same, which may be the
underlying mechanism for why the loss spike improves generalization, rather
than simply controlling the value of $\lambda_{\mathrm{max}}$.