Bayesian Inference for Consistent Predictions in Overparameterized
  Nonlinear Regression
Tomoya Wakayama
The remarkable generalization performance of overparameterized models has
challenged the conventional wisdom of statistical learning theory. While recent
theoretical studies have shed light on this behavior in linear models or
nonlinear classifiers, a comprehensive understanding of overparameterization in
nonlinear regression remains lacking. This paper explores the predictive
properties of overparameterized nonlinear regression within the Bayesian
framework, extending the methodology of adaptive prior based on the intrinsic
spectral structure of the data. We establish posterior contraction for
single-neuron models with Lipschitz continuous activation functions and for
generalized linear models, demonstrating that our approach achieves consistent
predictions in the overparameterized regime. Moreover, our Bayesian framework
allows for uncertainty estimation of the predictions. The proposed method is
validated through numerical simulations and a real data application, showcasing
its ability to achieve accurate predictions and reliable uncertainty estimates.
Our work advances the theoretical understanding of the blessing of
overparameterization and offers a principled Bayesian approach for prediction
in large nonlinear models.