Occam's Razor for Self Supervised Learning: What is Sufficient to Learn Good Representations?
This addresses the complexity and hyper-parameter sensitivity in SSL for practitioners in small to medium-scale settings, though it is incremental as it focuses on dataset size limitations.
The study found that for pretraining datasets up to a few hundred thousand samples, additional design choices in self-supervised learning (SSL) do not improve representation quality, simplifying deployment and validating theoretical studies.
Deep Learning is often depicted as a trio of data-architecture-loss. Yet, recent Self Supervised Learning (SSL) solutions have introduced numerous additional design choices, e.g., a projector network, positive views, or teacher-student networks. These additions pose two challenges. First, they limit the impact of theoretical studies that often fail to incorporate all those intertwined designs. Second, they slow-down the deployment of SSL methods to new domains as numerous hyper-parameters need to be carefully tuned. In this study, we bring forward the surprising observation that--at least for pretraining datasets of up to a few hundred thousands samples--the additional designs introduced by SSL do not contribute to the quality of the learned representations. That finding not only provides legitimacy to existing theoretical studies, but also simplifies the practitioner's path to SSL deployment in numerous small and medium scale settings. Our finding answers a long-lasting question: the often-experienced sensitivity to training settings and hyper-parameters encountered in SSL come from their design, rather than the absence of supervised guidance.