Lillama: Large Language Models Compression via Low-Rank Feature Distillation
This addresses the need for efficient compression of LLMs to reduce computational costs, though it is incremental as it builds on prior observations about low-rank activations.
The paper tackles the problem of compressing large language models by proposing Lillama, a method that uses low-rank feature distillation to compress models like Mixtral-8x7B, removing 10 billion parameters while retaining over 95% performance, and compressing Phi-2 3B by 40% with minimal calibration data.
Current LLM structured pruning methods typically involve two steps: (1) compression with calibration data and (2) costly continued pretraining on billions of tokens to recover lost performance. This second step is necessary as the first significantly impacts model accuracy. Prior research suggests pretrained Transformer weights aren't inherently low-rank, unlike their activations, which may explain this drop. Based on this observation, we propose Lillama, a compression method that locally distills activations with low-rank weights. Using SVD for initialization and a joint loss combining teacher and student activations, we accelerate convergence and reduce memory use with local gradient updates. Lillama compresses Mixtral-8x7B within minutes on a single A100 GPU, removing 10 billion parameters while retaining over 95% of its original performance. Phi-2 3B can be compressed by 40% with just 13 million calibration tokens, resulting in a small model that competes with recent models of similar size. The method generalizes well to non-transformer architectures, compressing Mamba-3B by 20% while maintaining 99% performance.