Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow
  Solutions in Scalar Networks and Beyond
Itai Kreisler Mor Shpigel Nacson Daniel Soudry Yair Carmon
Recent research shows that when Gradient Descent (GD) is applied to neural
networks, the loss almost never decreases monotonically. Instead, the loss
oscillates as gradient descent converges to its ''Edge of Stability'' (EoS).
Here, we find a quantity that does decrease monotonically throughout GD
training: the sharpness attained by the gradient flow solution (GFS)-the
solution that would be obtained if, from now until convergence, we train with
an infinitesimal step size. Theoretically, we analyze scalar neural networks
with the squared loss, perhaps the simplest setting where the EoS phenomena
still occur. In this model, we prove that the GFS sharpness decreases
monotonically. Using this result, we characterize settings where GD provably
converges to the EoS in scalar networks. Empirically, we show that GD
monotonically decreases the GFS sharpness in a squared regression model as well
as practical neural network architectures.