Randomly Initialized Networks Can Learn from Peer-to-Peer Consensus
This work isolates the role of self-distillation in learning dynamics, providing a minimal baseline for understanding representation learning in self-supervised methods.
The authors show that randomly initialized networks can learn useful representations through peer-to-peer consensus alone, without projectors, predictors, or pretext tasks, achieving non-trivial improvements over random baselines on downstream tasks.
In self-supervised learning, self-distilled methods have shown impressive performance, learning representations useful for downstream tasks and even displaying emergent properties. However, state-of-the-art methods usually rely on ensembles of complex mechanisms, with many design choices that are empirically motivated and not well understood. In this work, we explore the role of self-distillation within learning dynamics. Specifically, we isolate the effect of self-distillation by training a group of randomly initialized networks, removing all other common components such as projectors, predictors, and even pretext tasks. Our findings show that even this minimal setup can lead to learned representations with non-trivial improvements over a random baseline on downstream tasks. We also demonstrate how this effect varies with different hyperparameters and present a short analysis of what is being learned by the models under this setup.