Enhancing Trustworthiness with Mixed Precision: Benchmarks, Opportunities, and Challenges
This addresses the need for trustworthy model compression in safety-critical applications like finance and healthcare, though it is incremental as it builds on existing quantization methods.
The paper tackles the problem that quantization of large language models often neglects trustworthiness metrics, risking deployment in high-stakes domains, and develops a precision-ensemble voting approach that improves performance by up to 5.8% on these metrics.
Large language models (LLMs) have shown promising performance across various tasks. However, their autoregressive decoding process poses significant challenges for efficient deployment on existing AI hardware. Quantization alleviates memory and compute pressure by compressing weights, activations, and KV caches to low precisions while preserving generation quality. However, existing quantization frameworks typically focus on perplexity or classification accuracy, often omitting critical trustworthiness metrics. This gap introduces risks when applying quantized LLMs to downstream high-stakes domains such as finance and healthcare. In this work, we systematically investigate the impact of quantization on four trustworthiness metrics (adversarial robustness, fairness, machine ethics, and out-of-distribution robustness) and identify the instability across compression ratios and quantization methods. Building on these observations, we develop a novel precision-ensemble voting approach that leverages predictions from mixed-precision variants of the same model and consistently improves performance by up to $5.8\%$ on trustworthiness metrics. Our results highlight the importance of considering trustworthiness when developing model compression techniques and point to research opportunities at the intersection of compression and trustworthiness for safety-critical applications.