Spreads in Effective Learning Rates: The Perils of Batch Normalization
  During Early Training
Christian H. X. Ali Mehmeti-Göpel Michael Wand
Excursions in gradient magnitude pose a persistent challenge when training
deep networks. In this paper, we study the early training phases of deep
normalized ReLU networks, accounting for the induced scale invariance by
examining effective learning rates (LRs). Starting with the well-known fact
that batch normalization (BN) leads to exponentially exploding gradients at
initialization, we develop an ODE-based model to describe early training
dynamics. Our model predicts that in the gradient flow, effective LRs will
eventually equalize, aligning with empirical findings on warm-up training.
Using large LRs is analogous to applying an explicit solver to a stiff
non-linear ODE, causing overshooting and vanishing gradients in lower layers
after the first step. Achieving overall balance demands careful tuning of LRs,
depth, and (optionally) momentum. Our model predicts the formation of spreads
in effective LRs, consistent with empirical measurements. Moreover, we observe
that large spreads in effective LRs result in training issues concerning
accuracy, indicating the importance of controlling these dynamics. To further
support a causal relationship, we implement a simple scheduling scheme
prescribing uniform effective LRs across layers and confirm accuracy benefits.