SVD-Surgeon: Optimal Singular-Value Surgery for Large Language Model Compression
For practitioners deploying LLMs, SVD-Surgeon offers a training-free method to improve compression quality, though it is an incremental improvement over existing SVD-based approaches.
SVD-Surgeon improves low-rank compression of LLMs by applying the Optimal Brain Surgeon framework to singular values, enabling closed-form updates that compensate for truncated components. It enhances the perplexity-compression trade-off of SVD-LLM on OPT models and LLaMA 2-7B without retraining.
Large language models (LLMs) achieve remarkable performance across a wide range of tasks, but their deployment is constrained by substantial memory and compute requirements. Low-rank compression via singular value decomposition (SVD) is an effective remedy, but existing methods focus on how to factorize and which components to keep. We introduce SVD-Surgeon, a training-free method that brings the Optimal Brain Surgeon (OBS) framework to the singular-value basis. Treating each singular value as a parameter, it computes a closed-form update of the retained singular values that compensates, to second order in the model loss, for those removed by truncation. The same analysis yields a saliency for choosing which values to prune. As it operates directly on the singular-value factorization, SVD-Surgeon can be layered on top of existing SVD compressors. Applied to SVD-LLM, a leading SVD-based method, it improves the perplexity-compression trade-off on the OPT family and LLaMA 2-7B without any retraining.