On the Disconnect Between Theory and Practice of Overparametrized Neural
  Networks
Jonathan Wenger Felix Dangel Agustinus Kristiadi
The infinite-width limit of neural networks (NNs) has garnered significant
attention as a theoretical framework for analyzing the behavior of large-scale,
overparametrized networks. By approaching infinite width, NNs effectively
converge to a linear model with features characterized by the neural tangent
kernel (NTK). This establishes a connection between NNs and kernel methods, the
latter of which are well understood. Based on this link, theoretical benefits
and algorithmic improvements have been hypothesized and empirically
demonstrated in synthetic architectures. These advantages include faster
optimization, reliable uncertainty quantification and improved continual
learning. However, current results quantifying the rate of convergence to the
kernel regime suggest that exploiting these benefits requires architectures
that are orders of magnitude wider than they are deep. This assumption raises
concerns that practically relevant architectures do not exhibit behavior as
predicted via the NTK. In this work, we empirically investigate whether the
limiting regime either describes the behavior of large-width architectures used
in practice or is informative for algorithmic improvements. Our empirical
results demonstrate that this is not the case in optimization, uncertainty
quantification or continual learning. This observed disconnect between theory
and practice calls into question the practical relevance of the infinite-width
limit.