Why do Random Forests Work? Understanding Tree Ensembles as
  Self-Regularizing Adaptive Smoothers
Alicia Curth Alan Jeffares Mihaela van der Schaar
Despite their remarkable effectiveness and broad application, the drivers of
success underlying ensembles of trees are still not fully understood. In this
paper, we highlight how interpreting tree ensembles as adaptive and
self-regularizing smoothers can provide new intuition and deeper insight to
this topic. We use this perspective to show that, when studied as smoothers,
randomized tree ensembles not only make predictions that are quantifiably more
smooth than the predictions of the individual trees they consist of, but also
further regulate their smoothness at test-time based on the dissimilarity
between testing and training inputs. First, we use this insight to revisit,
refine and reconcile two recent explanations of forest success by providing a
new way of quantifying the conjectured behaviors of tree ensembles objectively
by measuring the effective degree of smoothing they imply. Then, we move beyond
existing explanations for the mechanisms by which tree ensembles improve upon
individual trees and challenge the popular wisdom that the superior performance
of forests should be understood as a consequence of variance reduction alone.
We argue that the current high-level dichotomy into bias- and
variance-reduction prevalent in statistics is insufficient to understand tree
ensembles -- because the prevailing definition of bias does not capture
differences in the expressivity of the hypothesis classes formed by trees and
forests. Instead, we show that forests can improve upon trees by three distinct
mechanisms that are usually implicitly entangled. In particular, we demonstrate
that the smoothing effect of ensembling can reduce variance in predictions due
to noise in outcome generation, reduce variability in the quality of the
learned function given fixed input data and reduce potential bias in learnable
functions by enriching the available hypothesis space.