Quantifying degeneracy in singular models via the learning coefficient
Edmund Lau Daniel Murfet Susan Wei
Deep neural networks (DNN) are singular statistical models which exhibit
complex degeneracies. In this work, we illustrate how a quantity known as the
\emph{learning coefficient} introduced in singular learning theory quantifies
precisely the degree of degeneracy in deep neural networks. Importantly, we
will demonstrate that degeneracy in DNN cannot be accounted for by simply
counting the number of "flat" directions. We propose a computationally scalable
approximation of a localized version of the learning coefficient using
stochastic gradient Langevin dynamics. To validate our approach, we demonstrate
its accuracy in low-dimensional models with known theoretical values.
Importantly, the local learning coefficient can correctly recover the ordering
of degeneracy between various parameter regions of interest. An experiment on
MNIST shows the local learning coefficient can reveal the inductive bias of
stochastic opitmizers for more or less degenerate critical points.