SiLQ: Simple Large Language Model Quantization-Aware Training
This work addresses the challenge of efficient model deployment for users needing lower latency and cost, though it appears incremental as it builds on existing quantization techniques.
The paper tackles the problem of quantizing large language models to reduce inference costs while maintaining accuracy, and demonstrates a simple quantization-aware training method that outperforms leading methods by large margins on modern benchmarks with less than 0.1% increase in training budget.
Large language models can be quantized to reduce inference time latency, model size, and energy consumption, thereby delivering a better user experience at lower cost. A challenge exists to deliver quantized models with minimal loss of accuracy in reasonable time, and in particular to do so without requiring mechanisms incompatible with specialized inference accelerators. Here, we demonstrate a simple, end-to-end quantization-aware training approach that, with an increase in total model training budget of less than 0.1%, outperforms the leading published quantization methods by large margins on several modern benchmarks, with both base and instruct model variants. The approach easily generalizes across different model architectures, can be applied to activations, cache, and weights, and requires the introduction of no additional operations to the model other than the quantization itself.