Learning Capacity: A Measure of the Effective Dimensionality of a Model
Daiwei Chen Weikai Chang Pratik Chaudhari
We exploit a formal correspondence between thermodynamics and inference,
where the number of samples can be thought of as the inverse temperature, to
define a "learning capacity'' which is a measure of the effective
dimensionality of a model. We show that the learning capacity is a tiny
fraction of the number of parameters for many deep networks trained on typical
datasets, depends upon the number of samples used for training, and is
numerically consistent with notions of capacity obtained from the PAC-Bayesian
framework. The test error as a function of the learning capacity does not
exhibit double descent. We show that the learning capacity of a model saturates
at very small and very large sample sizes; this provides guidelines, as to
whether one should procure more data or whether one should search for new
architectures, to improve performance. We show how the learning capacity can be
used to understand the effective dimensionality, even for non-parametric models
such as random forests and $k$-nearest neighbor classifiers.