Litespark Technical Report: High-Throughput, Energy-Efficient LLM Training Framework
This addresses inefficiencies in LLM training for AI researchers and practitioners, though it appears incremental as it builds on existing transformer optimizations.
The paper tackles the problem of long training times and high energy consumption in LLM training by introducing Litespark, a framework that achieves 2x-6x throughput improvement and 55%-83% energy reduction on Llama models.
Training Large Language Models (LLMs) is plagued by long training times and massive energy consumption, with modern models requiring months of computation and gigawatt-hours of electricity. In light of these challenges,we introduce Litespark, a novel pre-training framework that addresses these inefficiencies through targeted optimizations to transformer attention and MLP layers. Our approach combines architectural improvements with algorithmic enhancements to maximize Model FLOPs Utilization (MFU) while maintaining compatibility with standard transformer implementations. Comprehensive benchmarking on 3B and 30B parameter Llama models using the SlimPajama-627B dataset demonstrates substantial performance gains: 2x-6x training throughput improvement and $55\%-83$% energy consumption reduction across multi-node H200 GPU clusters. These optimizations are model- and hardware-agnostic, enabling broad applicability across transformer architectures and extending to post-training phases including supervised fine-tuning and direct preference optimization.