Cherry on Top: Parameter Heterogeneity and Quantization in Large Language Models
This addresses the challenge of efficient model deployment for AI practitioners by enabling aggressive quantization without significant performance loss, though it is incremental as it builds on existing quantization techniques.
The paper tackles the problem of parameter heterogeneity in large language models by identifying a small subset of influential 'cherry' parameters and proposes CherryQ, a quantization method that preserves these in high precision while aggressively quantizing others, resulting in a 3-bit quantized model achieving competitive performance with 16-bit counterparts.
This paper reveals the phenomenon of parameter heterogeneity in large language models (LLMs). We find that a small subset of "cherry" parameters exhibit a disproportionately large influence on model performance, while the vast majority of parameters have minimal impact. This heterogeneity is found to be prevalent across different model families, scales, and types. Motivated by this observation, we propose CherryQ, a novel quantization method that unifies the optimization of mixed-precision parameters. CherryQ identifies and preserves the critical cherry parameters in high precision while aggressively quantizing the remaining parameters to low precision. Extensive experiments demonstrate the effectiveness of CherryQ. CherryQ outperforms existing quantization approaches in terms of perplexity and downstream task performance. Notably, our 3-bit quantized Vicuna-1.5 exhibits competitive performance compared to their 16-bit counterparts.