Does Compression Preserve Uncertainty? A Unified Benchmark for Quantized and Sparse LLMs via Conformal Prediction
For practitioners deploying compressed LLMs in safety-critical applications, this work highlights that accuracy-only evaluation is insufficient and advocates for uncertainty-aware benchmarking.
The paper investigates whether compression techniques (quantization and pruning) preserve the uncertainty quantification ability of LLMs, using conformal prediction across 12 models and 5 tasks. It finds that compression often decouples accuracy from uncertainty, larger models handle compression better, and uncertainty inflation is threshold-like.
Model compression techniques such as quantization and pruning are widely used to reduce the deployment cost of large language models (LLMs), with existing evaluations focusing almost exclusively on accuracy preservation. However, in safety-critical applications, a model's ability to reliably quantify its own uncertainty is equally important. We ask: does compression preserve this ability? To answer this question, we benchmark 12 LLMs under various compression configurations across five NLP tasks, using conformal prediction to provide a rigorous, distribution-free measure of uncertainty. Our experiments reveal that: (I) compression frequently decouples accuracy from uncertainty; (II) larger models absorb compression-induced uncertainty far more effectively than smaller ones; and (III) uncertainty inflation is often threshold-like rather than gradual. These results suggest that accuracy-only evaluation is insufficient for assessing the deployment readiness of compressed LLMs, and that uncertainty-aware benchmarking should be a standard component of model compression pipelines.