Butterfly Effects of SGD Noise: Error Amplification in Behavior Cloning
  and Autoregression
Adam Block Dylan J. Foster Akshay Krishnamurthy Max Simchowitz Cyril Zhang
This work studies training instabilities of behavior cloning with deep neural
networks. We observe that minibatch SGD updates to the policy network during
training result in sharp oscillations in long-horizon rewards, despite
negligibly affecting the behavior cloning loss. We empirically disentangle the
statistical and computational causes of these oscillations, and find them to
stem from the chaotic propagation of minibatch SGD noise through unstable
closed-loop dynamics. While SGD noise is benign in the single-step action
prediction objective, it results in catastrophic error accumulation over long
horizons, an effect we term gradient variance amplification (GVA). We show that
many standard mitigation techniques do not alleviate GVA, but find an
exponential moving average (EMA) of iterates to be surprisingly effective at
doing so. We illustrate the generality of this phenomenon by showing the
existence of GVA and its amelioration by EMA in both continuous control and
autoregressive language generation. Finally, we provide theoretical vignettes
that highlight the benefits of EMA in alleviating GVA and shed light on the
extent to which classical convex models can help in understanding the benefits
of iterate averaging in deep learning.