Deconstructing the Goldilocks Zone of Neural Network Initialization
Artem Vysogorets Anna Dawid Julia Kempe
The second-order properties of the training loss have a massive impact on the
optimization dynamics of deep learning models. Fort & Scherlis (2019)
discovered that a high positive curvature and local convexity of the loss
Hessian are associated with highly trainable initial points located in a region
coined the "Goldilocks zone". Only a handful of subsequent studies touched upon
this relationship, so it remains largely unexplained. In this paper, we present
a rigorous and comprehensive analysis of the Goldilocks zone for homogeneous
neural networks. In particular, we derive the fundamental condition resulting
in non-zero positive curvature of the loss Hessian and argue that it is only
incidentally related to the initialization norm, contrary to prior beliefs.
Further, we relate high positive curvature to model confidence, low initial
loss, and a previously unknown type of vanishing cross-entropy loss gradient.
To understand the importance of positive curvature for trainability of deep
networks, we optimize both fully-connected and convolutional architectures
outside the Goldilocks zone and analyze the emergent behaviors. We find that
strong model performance is not necessarily aligned with the Goldilocks zone,
which questions the practical significance of this concept.