DeepTwist: Learning Model Compression via Occasional Weight Distortion
This work addresses the problem of reducing hardware resource requirements for machine learning models, offering a simpler approach to compression that is incremental over existing techniques.
The paper tackles the challenge of high design complexity in model compression by proposing DeepTwist, a framework that occasionally distorts weights without altering training algorithms, which significantly improves compression rates for pruning, quantization, and low-rank approximation while reducing retraining and hyper-parameter search efforts.
Model compression has been introduced to reduce the required hardware resources while maintaining the model accuracy. Lots of techniques for model compression, such as pruning, quantization, and low-rank approximation, have been suggested along with different inference implementation characteristics. Adopting model compression is, however, still challenging because the design complexity of model compression is rapidly increasing due to additional hyper-parameters and computation overhead in order to achieve a high compression ratio. In this paper, we propose a simple and efficient model compression framework called DeepTwist which distorts weights in an occasional manner without modifying the underlying training algorithms. The ideas of designing weight distortion functions are intuitive and straightforward given formats of compressed weights. We show that our proposed framework improves compression rate significantly for pruning, quantization, and low-rank approximation techniques while the efforts of additional retraining and/or hyper-parameter search are highly reduced. Regularization effects of DeepTwist are also reported.