Does Double Descent Occur in Self-Supervised Learning?
Alisia Lupidi Yonatan Gideoni Dulhan Jayalath
Most investigations into double descent have focused on supervised models
while the few works studying self-supervised settings find a surprising lack of
the phenomenon. These results imply that double descent may not exist in
self-supervised models. We show this empirically using a standard and linear
autoencoder, two previously unstudied settings. The test loss is found to have
either a classical U-shape or to monotonically decrease instead of exhibiting a
double-descent curve. We hope that further work on this will help elucidate the
theoretical underpinnings of this phenomenon.