Training Dynamics Impact Post-Training Quantization Robustness
This work addresses the problem of maintaining model efficiency through quantization for practitioners deploying large language models, though it appears incremental in nature.
The paper investigates how training dynamics affect post-training quantization robustness in large language models, finding that quantization errors are driven by complex interactions between learning rates and other hyperparameters rather than dataset scale. The researchers demonstrate that strategic hyperparameter interventions can improve quantization quality at scale, challenging the assumption that larger datasets inherently compromise quantization effectiveness.
While post-training quantization is widely adopted for efficient deployment of large language models, the mechanisms underlying quantization robustness remain unclear. We conduct a comprehensive analysis of quantization degradation across open-source language model training trajectories up to 32B parameters and 15T training tokens to accurately assess the relationship between training dynamics and quantization performance. Our key finding is that quantization errors in large-scale training runs are driven by a complex interplay between learning rate and other training hyperparameters. Specifically, once learning rates decay, validation loss and quantization error diverge, largely independent of training data scale. To investigate interventions on the training dynamics and identify specific configurations that can modulate quantization robustness favorably, we train our own models in controlled experiments up to 100B tokens. Our results challenge the assumption that increasing dataset scale inherently compromises quantization effectiveness, demonstrating instead that strategic training hyperparameter interventions can improve quantization quality at scale.