Mixtraining: A Better Trade-Off Between Compute and Performance
This addresses efficiency and performance issues in resource-constrained settings for machine learning practitioners, though it is incremental as it builds on existing SSL-SL strategies.
The paper tackles the trade-off between computation and performance in combining self-supervised and supervised learning by proposing MixTraining, which interleaves these phases to achieve an 8.81% absolute accuracy gain on TinyImageNet while accelerating training by up to 1.29x.
Incorporating self-supervised learning (SSL) before standard supervised learning (SL) has become a widely used strategy to enhance model performance, particularly in data-limited scenarios. However, this approach introduces a trade-off between computation and performance: while SSL helps with representation learning, it requires a separate, often time-consuming training phase, increasing computational overhead and limiting efficiency in resource-constrained settings. To address these challenges, we propose MixTraining, a novel framework that interleaves several SSL and SL epochs within a unified mixtraining training phase, featuring a smooth transition between two learning objectives. MixTraining enhances synergy between SSL and SL for improved accuracy and consolidates shared computation steps to reduce computation overhead. MixTraining is versatile and applicable to both single-task and multi-task learning scenarios. Extensive experiments demonstrate that MixTraining offers a superior compute-performance trade-off compared to conventional pipelines, achieving an 8.81% absolute accuracy gain (18.89% relative accuracy gain) on the TinyImageNet dataset while accelerating training by up to 1.29x with the ViT-Tiny model.