LLMQ: Efficient Lower-Precision Pretraining for Consumer GPUs
This work addresses the challenge of making large language model training accessible on affordable hardware for researchers and developers with limited resources, though it is incremental as it optimizes existing methods rather than introducing new paradigms.
The authors tackled the problem of training medium-sized language models on consumer GPUs with limited memory and slow communication by developing LLMQ, an efficient CUDA/C++ implementation that enables training a 7B model on a single 16GB GPU or a 32B model on four RTX 4090s while maintaining 50% FLOP utilization with 8-bit precision.
We present LLMQ, an end-to-end CUDA/C++ implementation for medium-sized language-model training, e.g. 3B to 32B parameters, on affordable, commodity GPUs. These devices are characterized by low memory availability and slow communication compared to datacentre-grade GPUs. Consequently, we showcase a range of optimizations that target these bottlenecks, including activation checkpointing, offloading, and copy-engine based collectives. LLMQ is able to train or fine-tune a 7B model on a single 16GB mid-range gaming card, or a 32B model on a workstation equipped with 4 RTX 4090s. This is achieved while executing a standard 8-bit training pipeline, without additional algorithmic approximations, and maintaining FLOP utilization of around 50%. The efficiency of LLMQ rivals that of production-scale systems on much more expensive cloud-grade GPUs.