Contrasting the landscape of contrastive and non-contrastive learning
This work addresses a fundamental issue in unsupervised learning for researchers, revealing limitations in non-contrastive methods that could impact algorithm design and theoretical understanding.
The paper investigates non-contrastive learning in unsupervised feature learning, showing that even on simple data models, these losses have many non-collapsed bad minima and training fails to avoid them, challenging the assumption that avoiding collapsed solutions ensures good representations.
A lot of recent advances in unsupervised feature learning are based on designing features which are invariant under semantic data augmentations. A common way to do this is contrastive learning, which uses positive and negative samples. Some recent works however have shown promising results for non-contrastive learning, which does not require negative samples. However, the non-contrastive losses have obvious "collapsed" minima, in which the encoders output a constant feature embedding, independent of the input. A folk conjecture is that so long as these collapsed solutions are avoided, the produced feature representations should be good. In our paper, we cast doubt on this story: we show through theoretical results and controlled experiments that even on simple data models, non-contrastive losses have a preponderance of non-collapsed bad minima. Moreover, we show that the training process does not avoid these minima.