Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models
This work addresses the practical need for efficient model compression in NLP, though it is incremental as it builds on existing offline compression approaches.
The paper tackles the problem of compressing large transformer language models for memory-constrained devices by proposing an autoencoder-based offline compression method that does not require fine-tuning, and it significantly outperforms existing factorization-based methods in experiments on various NLP tasks.
Recent transformer language models achieve outstanding results in many natural language processing (NLP) tasks. However, their enormous size often makes them impractical on memory-constrained devices, requiring practitioners to compress them to smaller networks. In this paper, we explore offline compression methods, meaning computationally-cheap approaches that do not require further fine-tuning of the compressed model. We challenge the classical matrix factorization methods by proposing a novel, better-performing autoencoder-based framework. We perform a comprehensive ablation study of our approach, examining its different aspects over a diverse set of evaluation settings. Moreover, we show that enabling collaboration between modules across layers by compressing certain modules together positively impacts the final model performance. Experiments on various NLP tasks demonstrate that our approach significantly outperforms commonly used factorization-based offline compression methods.