Stability and L2-penalty in Model Averaging
Hengkun Zhu Guohua Zou
Model averaging has received much attention in the past two decades, which
integrates available information by averaging over potential models. Although
various model averaging methods have been developed, there are few literatures
on the theoretical properties of model averaging from the perspective of
stability, and the majority of these methods constrain model weights to a
simplex. The aim of this paper is to introduce stability from statistical
learning theory into model averaging. Thus, we define the stability, asymptotic
empirical risk minimizer, generalization, and consistency of model averaging
and study the relationship among them. Our results indicate that stability can
ensure that model averaging has good generalization performance and consistency
under reasonable conditions, where consistency means model averaging estimator
can asymptotically minimize the mean squared prediction error. We also propose
a L2-penalty model averaging method without limiting model weights and prove
that it has stability and consistency. In order to reduce the impact of tuning
parameter selection, we use 10-fold cross-validation to select a candidate set
of tuning parameters and perform a weighted average of the estimators of model
weights based on estimation errors. The Monte Carlo simulation and an
illustrative application demonstrate the usefulness of the proposed method.