Evaluating the Impact of Compression Techniques on Task-Specific Performance of Large Language Models
This work addresses the computational cost problem for LLM users by highlighting evaluation and calibration issues in compression, though it is incremental as it builds on existing methods.
This study evaluated compression techniques like Magnitude Pruning, SparseGPT, and Wanda on the LLaMA-2-7B model, finding that while SparseGPT and Wanda maintained perplexity at 50% sparsity, they degraded downstream task performance, and it introduced Jensen-Shannon Divergence as a better metric and showed task-specific calibration data improves results.
Large language models (LLMs) offer powerful capabilities but incur substantial computational costs, driving the need for efficient compression techniques. This study evaluates the impact of popular compression methods - Magnitude Pruning, SparseGPT, and Wanda - on the LLaMA-2-7B model, focusing on the trade-offs between model size reduction, downstream task performance, and the role of calibration data. Our findings reveal that while SparseGPT and Wanda preserve perplexity even at 50% sparsity, they suffer significant degradation on downstream tasks, highlighting the inadequacy of perplexity as the sole evaluation metric. To address this, we introduce Jensen-Shannon (JS) Divergence as a more comprehensive metric that captures nuanced changes in model behavior post-compression. We further demonstrate that task-specific calibration data significantly enhances the downstream performance of compressed models compared to general calibration data. This research underscores the necessity for diverse evaluation metrics and careful calibration data selection to fully understand the complexities of LLM compression and its implications for practical applications.