On the curvature of the loss landscape
Alison Pouplin Hrittik Roy Sidak Pal Singh Georgios Arvanitidis
One of the main challenges in modern deep learning is to understand why such
over-parameterized models perform so well when trained on finite data. A way to
analyze this generalization concept is through the properties of the associated
loss landscape. In this work, we consider the loss landscape as an embedded
Riemannian manifold and show that the differential geometric properties of the
manifold can be used when analyzing the generalization abilities of a deep net.
In particular, we focus on the scalar curvature, which can be computed
analytically for our manifold, and show connections to several settings that
potentially imply generalization.