Correlated Noise in Epoch-Based Stochastic Gradient Descent:
  Implications for Weight Variances
Marcel Kühn Bernd Rosenow
Stochastic gradient descent (SGD) has become a cornerstone of neural network
optimization, yet the noise introduced by SGD is often assumed to be
uncorrelated over time, despite the ubiquity of epoch-based training. In this
work, we challenge this assumption and investigate the effects of epoch-based
noise correlations on the stationary distribution of discrete-time SGD with
momentum, limited to a quadratic loss. Our main contributions are twofold:
first, we calculate the exact autocorrelation of the noise for training in
epochs under the assumption that the noise is independent of small fluctuations
in the weight vector; second, we explore the influence of correlations
introduced by the epoch-based learning scheme on SGD dynamics. We find that for
directions with a curvature greater than a hyperparameter-dependent crossover
value, the results for uncorrelated noise are recovered. However, for
relatively flat directions, the weight variance is significantly reduced. We
provide an intuitive explanation for these results based on a crossover between
correlation times, contributing to a deeper understanding of the dynamics of
SGD in the presence of epoch-based noise correlations.