Numerical Optimizations for Weighted Low-rank Estimation on Language Model
This work addresses the need for more effective compression techniques for large language models, offering a practical improvement over standard SVD methods.
The paper tackles the problem of compressing Transformer-based language models by proposing a weighted low-rank estimation method that accounts for uneven parameter importance, achieving better performance than current state-of-the-art methods in evaluations.
Singular value decomposition (SVD) is one of the most popular compression methods that approximate a target matrix with smaller matrices. However, standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. The parameters of a trained neural network model may affect task performance unevenly, which suggests non-equal importance among the parameters. Compared to SVD, the decomposition method aware of parameter importance is the more practical choice in real cases. Unlike standard SVD, weighted value decomposition is a non-convex optimization problem that lacks a closed-form solution. We systematically investigated multiple optimization strategies to tackle the problem and examined our method by compressing Transformer-based language models. Further, we designed a metric to predict when the SVD may introduce a significant performance drop, for which our method can be a rescue strategy. The extensive evaluations demonstrate that our method can perform better than current SOTA methods in compressing Transformer-based language models.