UniComp: A Unified Evaluation of Large Language Model Compression via Pruning, Quantization and Distillation
This work addresses the need for comprehensive evaluation of compression techniques for deploying large language models, providing insights for researchers and practitioners, though it is incremental as it builds on existing compression methods.
The paper tackles the problem of evaluating large language model compression methods by introducing UniComp, a unified framework that assesses pruning, quantization, and distillation across performance, reliability, and efficiency dimensions, finding that quantization offers the best trade-off and task-specific calibration can improve reasoning in pruned models by up to 50%.
Model compression is increasingly essential for deploying large language models (LLMs), yet existing evaluations are limited in method coverage and focus primarily on knowledge-centric benchmarks. Thus, we introduce UniComp, a unified evaluation framework for comparing pruning, quantization, and knowledge distillation. UniComp evaluates compressed models along three dimensions: performance, reliability, and efficiency, using a diverse set of capability- and safety-oriented benchmarks together with a hardware-aware efficiency analysis. Through extensive evaluation of six compression techniques on modern LLMs across more than 40 datasets, we find that (i) compression exhibits a consistent knowledge bias, where knowledge-intensive tasks are relatively preserved while reasoning, multilingual, and instruction-following capabilities degrade substantially; (ii) quantization provides the best overall trade-off between retained performance and efficiency, whereas distillation yields strong runtime acceleration gains at high computational cost; and (iii) task-specific calibration can significantly improve the reasoning ability of pruned models by up to 50%.