On the Benefits of Over-parameterization for Out-of-Distribution
  Generalization
Yifan Hao Yong Lin Difan Zou Tong Zhang
In recent years, machine learning models have achieved success based on the
independently and identically distributed assumption. However, this assumption
can be easily violated in real-world applications, leading to the
Out-of-Distribution (OOD) problem. Understanding how modern over-parameterized
DNNs behave under non-trivial natural distributional shifts is essential, as
current theoretical understanding is insufficient. Existing theoretical works
often provide meaningless results for over-parameterized models in OOD
scenarios or even contradict empirical findings. To this end, we are
investigating the performance of the over-parameterized model in terms of OOD
generalization under the general benign overfitting conditions. Our analysis
focuses on a random feature model and examines non-trivial natural
distributional shifts, where the benign overfitting estimators demonstrate a
constant excess OOD loss, despite achieving zero excess in-distribution (ID)
loss. We demonstrate that in this scenario, further increasing the model's
parameterization can significantly reduce the OOD loss. Intuitively, the
variance term of ID loss remains low due to orthogonality of long-tail
features, meaning overfitting noise during training generally doesn't raise
testing loss. However, in OOD cases, distributional shift increases the
variance term. Thankfully, the inherent shift is unrelated to individual x,
maintaining the orthogonality of long-tail features. Expanding the hidden
dimension can additionally improve this orthogonality by mapping the features
into higher-dimensional spaces, thereby reducing the variance term. We further
show that model ensembles also improve OOD loss, akin to increasing model
capacity. These insights explain the empirical phenomenon of enhanced OOD
generalization through model ensembles, supported by consistent simulations
with theoretical results.