Beyond Activation Alignment:The Alignment-Diversity Tradeoff in Task-Aware LLM Quantization
For practitioners deploying LLMs under memory constraints, TASA provides a method to achieve higher accuracy at lower bit-widths by addressing the overlooked role of calibration data in task-aware quantization.
The paper identifies the Perplexity Illusion and Alignment-Diversity Tradeoff in LLM quantization, and proposes TASA, a two-level framework that improves mixed-precision quantization by optimizing calibration-data composition and bit allocation. On LLaMA-3-8B, TASA achieves 3.5-bit models that match or surpass 4-bit baselines, with over 20 absolute point improvement on GSM8K.
Mixed-precision quantization (MPQ) has become a key technique for deploying large language models under stringent memory and compute constraints. We first identify a phenomenon that we term the Perplexity Illusion: layers ranked as important by perplexity-based sensitivity show little rank correlation with those that are most influential for complex reasoning performance, with Kendall $τ\approx 0$ in our analysis. We further reveal an Alignment-Diversity Tradeoff: using only target-task calibration data can degrade post-quantization performance, whereas incorporating general-domain data stabilizes sensitivity estimation and improves robustness across tasks. Based on these observations, we propose TASA (Task-Aware Sensitivity Analysis), a two-level framework that jointly optimizes calibration-data composition and mixed-precision bit allocation. Specifically, TASA searches for a calibration-data mixture using a training-free gradient-trace alignment criterion, and then aggregates perplexity and reasoning-oriented sensitivity signals to guide both inter-layer and intra-layer bit allocation. Experiments on LLaMA-3-8B and Qwen2.5-7B reveal a precision inversion: appropriately allocated 3.5-bit models can match or surpass less task-aware 4-bit baselines. At an average precision of 3.5 bits, TASA matches or outperforms several competitive 4-bit uniform baselines in aggregate accuracy, and improves over the strongest W3 baseline on GSM8K by more than 20 absolute points on LLaMA-3-8B. These results show that calibration-data composition substantially affects task-sensitive quantization, a factor underexplored in prior work.