How many Neurons do we need? A refined Analysis for Shallow Networks
  trained with Gradient Descent
Mike Nguyen Nicole Mücke
We analyze the generalization properties of two-layer neural networks in the
neural tangent kernel (NTK) regime, trained with gradient descent (GD). For
early stopped GD we derive fast rates of convergence that are known to be
minimax optimal in the framework of non-parametric regression in reproducing
kernel Hilbert spaces. On our way, we precisely keep track of the number of
hidden neurons required for generalization and improve over existing results.
We further show that the weights during training remain in a vicinity around
initialization, the radius being dependent on structural assumptions such as
degree of smoothness of the regression function and eigenvalue decay of the
integral operator associated to the NTK.