A Survey on Statistical Theory of Deep Learning: Approximation, Training
  Dynamics, and Generative Models
Namjoon Suh Guang Cheng
In this article, we review the literature on statistical theories of neural
networks from three perspectives. In the first part, results on excess risks
for neural networks are reviewed in the nonparametric framework of regression
or classification. These results rely on explicit constructions of neural
networks, leading to fast convergence rates of excess risks, in that tools from
the approximation theory are adopted. Through these constructions, the width
and depth of the networks can be expressed in terms of sample size, data
dimension, and function smoothness. Nonetheless, their underlying analysis only
applies to the global minimizer in the highly non-convex landscape of deep
neural networks. This motivates us to review the training dynamics of neural
networks in the second part. Specifically, we review papers that attempt to
answer ``how the neural network trained via gradient-based methods finds the
solution that can generalize well on unseen data.'' In particular, two
well-known paradigms are reviewed: the Neural Tangent Kernel (NTK) paradigm,
and Mean-Field (MF) paradigm. In the last part, we review the most recent
theoretical advancements in generative models including Generative Adversarial
Networks (GANs), diffusion models, and in-context learning (ICL) in the Large
Language Models (LLMs). The former two models are known to be the main pillars
of the modern generative AI era, while ICL is a strong capability of LLMs in
learning from a few examples in the context. Finally, we conclude the paper by
suggesting several promising directions for deep learning theory.