BitSnap: Checkpoint Sparsification and Quantization in LLM Training
This addresses storage, memory, and fault tolerance issues for researchers and engineers training LLMs, representing a strong domain-specific optimization rather than a fundamental breakthrough.
The paper tackles the problem of inefficient checkpoint saving and loading in large language model training by proposing an adaptive checkpoint sparsification and quantization method, achieving a 16x compression ratio without accuracy loss and a 2x compression ratio with minimal precision loss.
As large language models (LLMs) continue to grow in size and complexity, efficient checkpoint saving\&loading has become crucial for managing storage, memory usage, and fault tolerance in LLM training. The current works do not comprehensively take into account the optimization of these several aspects. This paper proposes a novel checkpoint sparsification and quantization method that adapts dynamically to different training stages and model architectures. We present a comprehensive analysis of existing lossy and lossless compression techniques, identify current limitations, and introduce our adaptive approach that balances compression ratio, speed, and precision impact throughout the training process. Experiments on different sizes of LLMs demonstrate that our bitmask-based sparsification method achieves 16x compression ratio without compromising model accuracy. Additionally, the cluster-based quantization method achieves 2x compression ratio with little precision loss.