Zero-Shot Dynamic Quantization for Transformer Inference
This addresses the challenge of efficient inference for NLP models, though it appears incremental as it builds on existing quantization techniques.
The paper tackles the problem of accuracy loss when quantizing BERT-like models to 8-bit integers by introducing a run-time method that eliminates the need for training modifications or calibration steps, showing its usefulness on several NLP tasks.
We introduce a novel run-time method for significantly reducing the accuracy loss associated with quantizing BERT-like models to 8-bit integers. Existing methods for quantizing models either modify the training procedure,or they require an additional calibration step to adjust parameters that also requires a selected held-out dataset. Our method permits taking advantage of quantization without the need for these adjustments. We present results on several NLP tasks demonstrating the usefulness of this technique.