Beyond Implicit Bias: The Insignificance of SGD Noise in Online Learning
Nikhil Vyas Depen Morwani Rosie Zhao Gal Kaplun Sham Kakade Boaz Barak
The success of SGD in deep learning has been ascribed by prior works to the
implicit bias induced by high learning rate or small batch size ("SGD noise").
While prior works that focused on offline learning (i.e., multiple-epoch
training), we study the impact of SGD noise on online (i.e., single epoch)
learning. Through an extensive empirical analysis of image and language data,
we demonstrate that large learning rate and small batch size do not confer any
implicit bias advantages in online learning. In contrast to offline learning,
the benefits of SGD noise in online learning are strictly computational,
facilitating larger or more cost-effective gradient steps. Our work suggests
that SGD in the online regime can be construed as taking noisy steps along the
"golden path" of the noiseless gradient flow algorithm. We provide evidence to
support this hypothesis by conducting experiments that reduce SGD noise during
training and by measuring the pointwise functional distance between models
trained with varying SGD noise levels, but at equivalent loss values. Our
findings challenge the prevailing understanding of SGD and offer novel insights
into its role in online learning.