Simple Distillation Baselines for Improving Small Self-supervised Models
This work addresses the performance gap for small self-supervised models in machine learning, though it is incremental as it builds on existing distillation techniques.
The paper tackles the problem of improving small self-supervised models, which lag behind large ones, by introducing simple distillation baselines called SimDis, resulting in a new state-of-the-art for offline distillation and competitive performance with low overhead for online distillation.
While large self-supervised models have rivalled the performance of their supervised counterparts, small models still struggle. In this report, we explore simple baselines for improving small self-supervised models via distillation, called SimDis. Specifically, we present an offline-distillation baseline, which establishes a new state-of-the-art, and an online-distillation baseline, which achieves similar performance with minimal computational overhead. We hope these baselines will provide useful experience for relevant future research. Code is available at: https://github.com/JindongGu/SimDis/