Scaling Laws for Post Training Quantized Large Language Models
This work addresses the problem of unpredictable compression quality for practitioners using post-training quantization of LLMs, though it is incremental as it builds on existing scaling laws.
The study tackled the unpredictability of large language model performance after post-training weight quantization by conducting an empirical analysis across multiple models and quantization techniques, resulting in a statistical model that reasonably predicts quantized performance based on local loss landscape characteristics.
Generalization abilities of well-trained large language models (LLMs) are known to scale predictably as a function of model size. In contrast to the existence of practical scaling laws governing pre-training, the quality of LLMs after post-training compression remains highly unpredictable, often requiring case-by-case validation in practice. In this work, we attempted to close this gap for post-training weight quantization of LLMs by conducting a systematic empirical study on multiple LLM families quantized to numerous low-precision tensor data types using popular weight quantization techniques. We identified key scaling factors pertaining to characteristics of the local loss landscape, based on which the performance of quantized LLMs can be reasonably well predicted by a statistical model.