Accurate INT8 Training Through Dynamic Block-Level Fallback
This addresses training cost reduction for AI practitioners using modern Transformers, but it is incremental as it builds on existing block-level quantization methods.
The paper tackles the challenge of low-bit INT8 training for modern Transformer variants with GLU units, which suffer from activation outliers, by proposing Fallback Quantization that dynamically switches blocks from 8-bit to 16-bit precision. The result is a robust method achieving a 1.57x end-to-end training speedup on RTX4090 GPUs.
Transformer models have achieved remarkable success across various AI applications but face significant training costs. Low-bit training, such as INT8 training, can leverage computational units with higher throughput, and has already demonstrated its effectiveness on GPT2 models with block-level quantization. However, it struggles with modern Transformer variants incorporating GLU units. This is because those variants demonstrate complex distributions of activation outliers. To address the challenge, we propose Fallback Quantization, implementing mixed-precision GEMM that dynamically falls back 8-bit to 16-bit for activation blocks containing outliers. Experiments show that our approach is robustly competent in both fine-tuning and pretraining settings. Moreover, our method achieves a 1.57x end-to-end training speedup on RTX4090 GPUs.