Universally Slimmable Networks and Improved Training Techniques
This work addresses the need for flexible and efficient neural network deployment across various computational constraints, offering a systematic approach to optimize accuracy-efficiency trade-offs, though it is incremental as it builds upon existing slimmable network concepts.
The authors tackled the problem of neural networks needing to adaptively trade off accuracy and efficiency at runtime by proposing universally slimmable networks (US-Nets) that can execute at arbitrary widths, and they introduced improved training techniques like the sandwich rule and inplace distillation, resulting in enhanced performance on ImageNet classification, image super-resolution, and deep reinforcement learning tasks compared to baselines.
Slimmable networks are a family of neural networks that can instantly adjust the runtime width. The width can be chosen from a predefined widths set to adaptively optimize accuracy-efficiency trade-offs at runtime. In this work, we propose a systematic approach to train universally slimmable networks (US-Nets), extending slimmable networks to execute at arbitrary width, and generalizing to networks both with and without batch normalization layers. We further propose two improved training techniques for US-Nets, named the sandwich rule and inplace distillation, to enhance training process and boost testing accuracy. We show improved performance of universally slimmable MobileNet v1 and MobileNet v2 on ImageNet classification task, compared with individually trained ones and 4-switch slimmable network baselines. We also evaluate the proposed US-Nets and improved training techniques on tasks of image super-resolution and deep reinforcement learning. Extensive ablation experiments on these representative tasks demonstrate the effectiveness of our proposed methods. Our discovery opens up the possibility to directly evaluate FLOPs-Accuracy spectrum of network architectures. Code and models are available at: https://github.com/JiahuiYu/slimmable_networks