Weak Correlations as the Underlying Principle for Linearization of
  Gradient-Based Learning Systems
Ori Shem-Ur Yaron Oz
Deep learning models, such as wide neural networks, can be conceptualized as
nonlinear dynamical physical systems characterized by a multitude of
interacting degrees of freedom. Such systems in the infinite limit, tend to
exhibit simplified dynamics. This paper delves into gradient descent-based
learning algorithms, that display a linear structure in their parameter
dynamics, reminiscent of the neural tangent kernel. We establish this apparent
linearity arises due to weak correlations between the first and higher-order
derivatives of the hypothesis function, concerning the parameters, taken around
their initial values. This insight suggests that these weak correlations could
be the underlying reason for the observed linearization in such systems. As a
case in point, we showcase this weak correlations structure within neural
networks in the large width limit. Exploiting the relationship between
linearity and weak correlations, we derive a bound on deviations from linearity
observed during the training trajectory of stochastic gradient descent. To
facilitate our proof, we introduce a novel method to characterise the
asymptotic behavior of random tensors.