Provable Guarantees for Neural Networks via Gradient Feature Learning
Zhenmei Shi Junyi Wei Yingyu Liang
Neural networks have achieved remarkable empirical performance, while the
current theoretical analysis is not adequate for understanding their success,
e.g., the Neural Tangent Kernel approach fails to capture their key feature
learning ability, while recent analyses on feature learning are typically
problem-specific. This work proposes a unified analysis framework for two-layer
networks trained by gradient descent. The framework is centered around the
principle of feature learning from gradients, and its effectiveness is
demonstrated by applications in several prototypical problems, such as mixtures
of Gaussians and parity functions. The framework also sheds light on
interesting network learning phenomena such as feature learning beyond kernels
and the lottery ticket hypothesis.