Does Double Descent Occur in Self-Supervised Learning?
This addresses a gap in understanding double descent for self-supervised learning, which could inform theoretical developments in machine learning.
The paper investigates whether double descent occurs in self-supervised learning, finding that in linear autoencoder settings, test loss shows a U-shape or monotonic decrease instead of double descent, implying the phenomenon may not exist in these models.
Most investigations into double descent have focused on supervised models while the few works studying self-supervised settings find a surprising lack of the phenomenon. These results imply that double descent may not exist in self-supervised models. We show this empirically using a standard and linear autoencoder, two previously unstudied settings. The test loss is found to have either a classical U-shape or to monotonically decrease instead of exhibiting a double-descent curve. We hope that further work on this will help elucidate the theoretical underpinnings of this phenomenon.