The Interpolating Information Criterion for Overparameterized Models
Liam Hodgkinson Chris van der Heide Robert Salomone Fred Roosta Michael W. Mahoney
The problem of model selection is considered for the setting of interpolating
estimators, where the number of model parameters exceeds the size of the
dataset. Classical information criteria typically consider the large-data
limit, penalizing model size. However, these criteria are not appropriate in
modern settings where overparameterized models tend to perform well. For any
overparameterized model, we show that there exists a dual underparameterized
model that possesses the same marginal likelihood, thus establishing a form of
Bayesian duality. This enables more classical methods to be used in the
overparameterized setting, revealing the Interpolating Information Criterion, a
measure of model quality that naturally incorporates the choice of prior into
the model selection. Our new information criterion accounts for prior
misspecification, geometric and spectral properties of the model, and is
numerically consistent with known empirical and theoretical behavior in this
regime.