Cross-Layer Subspace Coupling for LLM Compression: A Unifying Framework and Its Empirical Limits
This paper identifies a fundamental flaw in cross-layer weight space reconstruction objectives for LLM compression, which is significant for researchers developing more effective compression techniques.
This paper unifies recent SVD-based LLM compression methods under a single optimization problem. While this unified approach improves weight reconstruction error by up to 46% on Pythia models, it paradoxically degrades downstream metrics like perplexity and accuracy compared to per-layer SVD LLM.
Recent SVD based compression methods for large language models like SVD LLM and Basis Sharing can be unified under one optimization problem. While mathematical proofs and tests on Pythia models show this unified approach improves weight reconstruction error by up to 46% percent it fails in practical tasks. Downstream metrics like perplexity and accuracy severely degrade compared to standard per layer SVD LLM. The authors explain this failure mechanistically. Although the bundle method mathematically couples adjacent layers the transformer residual stream actually decouples them during forward passes. Thus per layer optimality matters more than joint cross layer optimization. The paper concludes that weight space reconstruction is a flawed objective for cross layer compression and future methods must focus on per layer activation reconstruction instead.