Understanding the Double Descent Phenomenon in Deep Learning
Marc Lafon Alexandre Thomas
Combining empirical risk minimization with capacity control is a classical
strategy in machine learning when trying to control the generalization gap and
avoid overfitting, as the model class capacity gets larger. Yet, in modern deep
learning practice, very large over-parameterized models (e.g. neural networks)
are optimized to fit perfectly the training data and still obtain great
generalization performance. Past the interpolation point, increasing model
complexity seems to actually lower the test error.
  In this tutorial, we explain the concept of double descent and its
mechanisms. The first section sets the classical statistical learning framework
and introduces the double descent phenomenon. By looking at a number of
examples, section 2 introduces inductive biases that appear to have a key role
in double descent by selecting, among the multiple interpolating solutions, a
smooth empirical risk minimizer. Finally, section 3 explores the double descent
with two linear models, and gives other points of view from recent related
works.