Early Directional Convergence in Deep Homogeneous Neural Networks for
  Small Initializations
Akshay Kumar Jarvis Haupt
This paper studies the gradient flow dynamics that arise when training deep
homogeneous neural networks, starting with small initializations. The present
work considers neural networks that are assumed to have locally Lipschitz
gradients and an order of homogeneity strictly greater than two. This paper
demonstrates that for sufficiently small initializations, during the early
stages of training, the weights of the neural network remain small in norm and
approximately converge in direction along the Karush-Kuhn-Tucker (KKT) points
of the neural correlation function introduced in [1]. Additionally, for square
loss and under a separability assumption on the weights of neural networks, a
similar directional convergence of gradient flow dynamics is shown near certain
saddle points of the loss function.