Generalization Error Curves for Analytic Spectral Algorithms under
  Power-law Decay
Yicheng Li Weiye Gan Zuoqiang Shi Qian Lin
The generalization error curve of certain kernel regression method aims at
determining the exact order of generalization error with various source
condition, noise level and choice of the regularization parameter rather than
the minimax rate. In this work, under mild assumptions, we rigorously provide a
full characterization of the generalization error curves of the kernel gradient
descent method (and a large class of analytic spectral algorithms) in kernel
regression. Consequently, we could sharpen the near inconsistency of kernel
interpolation and clarify the saturation effects of kernel regression
algorithms with higher qualification, etc. Thanks to the neural tangent kernel
theory, these results greatly improve our understanding of the generalization
behavior of training the wide neural networks. A novel technical contribution,
the analytic functional argument, might be of independent interest.