A U-turn on Double Descent: Rethinking Parameter Counting in Statistical
  Learning
Alicia Curth Alan Jeffares Mihaela van der Schaar
Conventional statistical wisdom established a well-understood relationship
between model complexity and prediction error, typically presented as a
U-shaped curve reflecting a transition between under- and overfitting regimes.
However, motivated by the success of overparametrized neural networks, recent
influential work has suggested this theory to be generally incomplete,
introducing an additional regime that exhibits a second descent in test error
as the parameter count p grows past sample size n - a phenomenon dubbed double
descent. While most attention has naturally been given to the deep-learning
setting, double descent was shown to emerge more generally across non-neural
models: known cases include linear regression, trees, and boosting. In this
work, we take a closer look at evidence surrounding these more classical
statistical machine learning methods and challenge the claim that observed
cases of double descent truly extend the limits of a traditional U-shaped
complexity-generalization curve therein. We show that once careful
consideration is given to what is being plotted on the x-axes of their double
descent plots, it becomes apparent that there are implicitly multiple
complexity axes along which the parameter count grows. We demonstrate that the
second descent appears exactly (and only) when and where the transition between
these underlying axes occurs, and that its location is thus not inherently tied
to the interpolation threshold p=n. We then gain further insight by adopting a
classical nonparametric statistics perspective. We interpret the investigated
methods as smoothers and propose a generalized measure for the effective number
of parameters they use on unseen examples, using which we find that their
apparent double descent curves indeed fold back into more traditional convex
shapes - providing a resolution to tensions between double descent and
statistical intuition.