DSPNet: Towards Slimmable Pretrained Networks based on Discriminative Self-supervised Learning
This addresses the problem of resource-efficient deployment for practitioners by providing a more efficient pretraining method, though it is incremental as it builds on existing slimmable and SSL techniques.
The paper tackles the high computational cost of pretraining multiple self-supervised learning networks for different resource budgets by proposing DSPNet, which trains once and slims to various sizes, achieving comparable or improved performance on ImageNet while reducing training costs.
Self-supervised learning (SSL) has achieved promising downstream performance. However, when facing various resource budgets in real-world applications, it costs a huge computation burden to pretrain multiple networks of various sizes one by one. In this paper, we propose Discriminative-SSL-based Slimmable Pretrained Networks (DSPNet), which can be trained at once and then slimmed to multiple sub-networks of various sizes, each of which faithfully learns good representation and can serve as good initialization for downstream tasks with various resource budgets. Specifically, we extend the idea of slimmable networks to a discriminative SSL paradigm, by integrating SSL and knowledge distillation gracefully. We show comparable or improved performance of DSPNet on ImageNet to the networks individually pretrained one by one under the linear evaluation and semi-supervised evaluation protocols, while reducing large training cost. The pretrained models also generalize well on downstream detection and segmentation tasks. Code will be made public.