Compressing Language Models using Doped Kronecker Products
This work addresses compression for large NLP models, offering a novel approach to reduce memory usage while maintaining performance, though it is incremental as it builds on existing Kronecker Product techniques.
The paper tackles the problem of significant accuracy loss when applying Kronecker Products to compress large language models, proposing a doping method that adds a sparse overlay matrix to recover accuracy, achieving 25x compression with only 1.4% perplexity loss compared to higher losses from other methods.
Kronecker Products (KP) have been used to compress IoT RNN Applications by 15-38x compression factors, achieving better results than traditional compression methods. However when KP is applied to large Natural Language Processing tasks, it leads to significant accuracy loss (approx 26%). This paper proposes a way to recover accuracy otherwise lost when applying KP to large NLP tasks, by allowing additional degrees of freedom in the KP matrix. More formally, we propose doping, a process of adding an extremely sparse overlay matrix on top of the pre-defined KP structure. We call this compression method doped kronecker product compression. To train these models, we present a new solution to the phenomenon of co-matrix adaption (CMA), which uses a new regularization scheme called co matrix dropout regularization (CMR). We present experimental results that demonstrate compression of a large language model with LSTM layers of size 25 MB by 25x with 1.4% loss in perplexity score. At 25x compression, an equivalent pruned network leads to 7.9% loss in perplexity score, while HMD and LMF lead to 15% and 27% loss in perplexity score respectively.