Unveiling the Hessian's Connection to the Decision Boundary
Mahalakshmi Sabanayagam Freya Behrens Urte Adomaityte Anna Dawid
Understanding the properties of well-generalizing minima is at the heart of
deep learning research. On the one hand, the generalization of neural networks
has been connected to the decision boundary complexity, which is hard to study
in the high-dimensional input space. Conversely, the flatness of a minimum has
become a controversial proxy for generalization. In this work, we provide the
missing link between the two approaches and show that the Hessian top
eigenvectors characterize the decision boundary learned by the neural network.
Notably, the number of outliers in the Hessian spectrum is proportional to the
complexity of the decision boundary. Based on this finding, we provide a new
and straightforward approach to studying the complexity of a high-dimensional
decision boundary; show that this connection naturally inspires a new
generalization measure; and finally, we develop a novel margin estimation
technique which, in combination with the generalization measure, precisely
identifies minima with simple wide-margin boundaries. Overall, this analysis
establishes the connection between the Hessian and the decision boundary and
provides a new method to identify minima with simple wide-margin decision
boundaries.