Optimizing Large Language Model Training Using FP4 Quantization
This addresses the problem of high computational demands for researchers and practitioners training LLMs, offering an incremental improvement over existing quantization methods.
The paper tackles the challenge of training large language models (LLMs) with FP4 quantization to reduce computational costs, achieving accuracy comparable to BF16 and FP8 with minimal degradation, scaling to 13B-parameter models on 100B tokens.
The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates a mixed-precision training scheme and vector-wise quantization. Experimental results demonstrate that our FP4 framework achieves accuracy comparable to BF16 and FP8, with minimal degradation, scaling effectively to 13B-parameter LLMs trained on up to 100B tokens. With the emergence of next-generation hardware supporting FP4, our framework sets a foundation for efficient ultra-low precision training.