Improve Generalization Ability of Deep Wide Residual Network with A
  Suitable Scaling Factor
Songtao Tian Zixiong Yu
Deep Residual Neural Networks (ResNets) have demonstrated remarkable success
across a wide range of real-world applications. In this paper, we identify a
suitable scaling factor (denoted by $\alpha$) on the residual branch of deep
wide ResNets to achieve good generalization ability. We show that if $\alpha$
is a constant, the class of functions induced by Residual Neural Tangent Kernel
(RNTK) is asymptotically not learnable, as the depth goes to infinity. We also
highlight a surprising phenomenon: even if we allow $\alpha$ to decrease with
increasing depth $L$, the degeneration phenomenon may still occur. However,
when $\alpha$ decreases rapidly with $L$, the kernel regression with deep RNTK
with early stopping can achieve the minimax rate provided that the target
regression function falls in the reproducing kernel Hilbert space associated
with the infinite-depth RNTK. Our simulation studies on synthetic data and real
classification tasks such as MNIST, CIFAR10 and CIFAR100 support our
theoretical criteria for choosing $\alpha$.