Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs
For practitioners deploying large language models, AIR provides a more efficient compression method that reduces data requirements and improves performance trade-offs.
AIR is an SVD-based LLM compression framework that uses a backward-signal influence metric to guide low-rank approximations. It improves perplexity over SVD-LLM(W) by >18% at ≤60% parameter retention, matches its quality with ~90% less calibration data, and yields FLOP, memory, and latency gains.
We present Activation- and Influence-Aware Ranks (AIR), an SVD-based LLM compression framework that guides each weight matrix's low-rank approximation with a backward-signal influence metric. Starting from the activation-aware optimum of SVD-LLM(W), AIR runs a single closed-form alternating least squares (ALS) sweep that integrates influence element-wise under a monotone-descent guarantee. AIR is layer-local and composes orthogonally with end-to-end methods: alone it exceeds ACIP, and AIR+LoRA outperforms it further. AIR improves perplexity over SVD-LLM(W) by >18% at <=60% parameter retention, matches its quality with ~90% less calibration data, and turns parameter savings into FLOP, peak-memory, and per-token latency gains.