SANE: The phases of gradient descent through Sharpness Adjusted Number
  of Effective parameters
Lawrence Wang Stephen J. Roberts
Modern neural networks are undeniably successful. Numerous studies have
investigated how the curvature of loss landscapes can affect the quality of
solutions. In this work we consider the Hessian matrix during network training.
We reiterate the connection between the number of "well-determined" or
"effective" parameters and the generalisation performance of neural nets, and
we demonstrate its use as a tool for model comparison. By considering the local
curvature, we propose Sharpness Adjusted Number of Effective parameters (SANE),
a measure of effective dimensionality for the quality of solutions. We show
that SANE is robust to large learning rates, which represent learning regimes
that are attractive but (in)famously unstable. We provide evidence and
characterise the Hessian shifts across "loss basins" at large learning rates.
Finally, extending our analysis to deeper neural networks, we provide an
approximation to the full-network Hessian, exploiting the natural ordering of
neural weights, and use this approximation to provide extensive empirical
evidence for our claims.