LoRAP: Transformer Sub-Layers Deserve Differentiated Structured Compression for Large Language Models
This work addresses the memory and computational resource challenges of LLMs, offering a domain-specific compression technique that is incremental in nature.
The paper tackles the problem of reducing the parameter scale of large language models (LLMs) by proposing LoRAP, a mixed compression model that combines low-rank approximation for multi-head self-attention sub-layers and structured pruning for feed-forward network sub-layers, achieving superior performance over previous structured compression methods in evaluations on zero-shot perplexity and task classification under multiple compression ratios.
Large language models (LLMs) show excellent performance in difficult tasks, but they often require massive memories and computational resources. How to reduce the parameter scale of LLMs has become research hotspots. In this study, we make an important observation that the multi-head self-attention (MHA) sub-layer of Transformer exhibits noticeable low-rank structure, while the feed-forward network (FFN) sub-layer does not. With this regard, we design a mixed compression model, which organically combines Low-Rank matrix approximation And structured Pruning (LoRAP). For the MHA sub-layer, we propose an input activation weighted singular value decomposition method to strengthen the low-rank characteristic. Furthermore, we discover that the weight matrices in MHA sub-layer have different low-rank degrees. Thus, a novel parameter allocation scheme according to the discrepancy of low-rank degrees is devised. For the FFN sub-layer, we propose a gradient-free structured channel pruning method. During the pruning, we get an interesting finding that the least important 1% of parameter actually play a vital role in model performance. Extensive evaluations on zero-shot perplexity and zero-shot task classification indicate that our proposal is superior to previous structured compression rivals under multiple compression ratios.