One Network Doesn't Rule Them All: Moving Beyond Handcrafted Architectures in Self-Supervised Learning
This work addresses the need for better architecture design in SSL for researchers and practitioners, moving beyond incremental improvements by integrating architecture search into SSL objectives.
The paper tackles the problem of overlooking network architecture's role in self-supervised learning (SSL) by conducting a large-scale study with over 100 architecture variants across 11 downstream scenarios, showing no single network performs consistently well. It proposes learning architecture topologies in SSL, with results indicating that self-supervised architectures outperform handcrafted ones like ResNet18 and MobileNetV2 and compete with ResNet50 on benchmarks such as ImageNet-1K and iNat2021.
The current literature on self-supervised learning (SSL) focuses on developing learning objectives to train neural networks more effectively on unlabeled data. The typical development process involves taking well-established architectures, e.g., ResNet demonstrated on ImageNet, and using them to evaluate newly developed objectives on downstream scenarios. While convenient, this does not take into account the role of architectures which has been shown to be crucial in the supervised learning literature. In this work, we establish extensive empirical evidence showing that a network architecture plays a significant role in SSL. We conduct a large-scale study with over 100 variants of ResNet and MobileNet architectures and evaluate them across 11 downstream scenarios in the SSL setting. We show that there is no one network that performs consistently well across the scenarios. Based on this, we propose to learn not only network weights but also architecture topologies in the SSL regime. We show that "self-supervised architectures" outperform popular handcrafted architectures (ResNet18 and MobileNetV2) while performing competitively with the larger and computationally heavy ResNet50 on major image classification benchmarks (ImageNet-1K, iNat2021, and more). Our results suggest that it is time to consider moving beyond handcrafted architectures in SSL and start thinking about incorporating architecture search into self-supervised learning objectives.