Demystifying Lazy Training of Neural Networks from a Macroscopic
  Viewpoint
Yuqing Li Tao Luo Qixuan Zhou
In this paper, we advance the understanding of neural network training
dynamics by examining the intricate interplay of various factors introduced by
weight parameters in the initialization process. Motivated by the foundational
work of Luo et al. (J. Mach. Learn. Res., Vol. 22, Iss. 1, No. 71, pp
3327-3373), we explore the gradient descent dynamics of neural networks through
the lens of macroscopic limits, where we analyze its behavior as width $m$
tends to infinity. Our study presents a unified approach with refined
techniques designed for multi-layer fully connected neural networks, which can
be readily extended to other neural network architectures. Our investigation
reveals that gradient descent can rapidly drive deep neural networks to zero
training loss, irrespective of the specific initialization schemes employed by
weight parameters, provided that the initial scale of the output function
$\kappa$ surpasses a certain threshold. This regime, characterized as the
theta-lazy area, accentuates the predominant influence of the initial scale
$\kappa$ over other factors on the training behavior of neural networks.
Furthermore, our approach draws inspiration from the Neural Tangent Kernel
(NTK) paradigm, and we expand its applicability. While NTK typically assumes
that $\lim_{m\to\infty}\frac{\log \kappa}{\log m}=\frac{1}{2}$, and imposes
each weight parameters to scale by the factor $\frac{1}{\sqrt{m}}$, in our
theta-lazy regime, we discard the factor and relax the conditions to
$\lim_{m\to\infty}\frac{\log \kappa}{\log m}>0$. Similar to NTK, the behavior
of overparameterized neural networks within the theta-lazy regime trained by
gradient descent can be effectively described by a specific kernel. Through
rigorous analysis, our investigation illuminates the pivotal role of $\kappa$
in governing the training dynamics of neural networks.