Compression-aware Training of Neural Networks using Frank-Wolfe
This addresses the need for efficient model compression in deep learning, offering a novel method that avoids retraining and reduces resource usage, though it is incremental in the context of existing compression-aware approaches.
The paper tackles the problem of training neural networks that are robust to compression without retraining, achieving state-of-the-art performance in compression-aware training and reducing computational costs for low-rank decomposition compared to nuclear-norm regularization.
Many existing Neural Network pruning approaches rely on either retraining or inducing a strong bias in order to converge to a sparse solution throughout training. A third paradigm, 'compression-aware' training, aims to obtain state-of-the-art dense models that are robust to a wide range of compression ratios using a single dense training run while also avoiding retraining. We propose a framework centered around a versatile family of norm constraints and the Stochastic Frank-Wolfe (SFW) algorithm that encourage convergence to well-performing solutions while inducing robustness towards convolutional filter pruning and low-rank matrix decomposition. Our method is able to outperform existing compression-aware approaches and, in the case of low-rank matrix decomposition, it also requires significantly less computational resources than approaches based on nuclear-norm regularization. Our findings indicate that dynamically adjusting the learning rate of SFW, as suggested by Pokutta et al. (2020), is crucial for convergence and robustness of SFW-trained models and we establish a theoretical foundation for that practice.