COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression
This work addresses the need for efficient compression of large Transformer models in resource-constrained environments, offering a novel method that improves accuracy and flexibility without iterative optimization.
The paper tackles the problem of compressing Transformer models post-training by proposing COMPOT, a training-free framework that uses a small calibration dataset to estimate sparse weight factorization and dynamically allocates compression rates across layers, achieving a superior quality-compression trade-off over existing low-rank and sparse baselines.
Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, eliminating iterative optimization. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression. Code is available $\href{https://github.com/mts-ai/COMPOT}{here}$.