EcoSpa: Efficient Transformer Training with Coupled Sparsity
This addresses efficiency challenges for AI practitioners using transformers, offering accessible improvements on commodity hardware, but it is incremental as it builds on existing sparse training methods.
The paper tackles the problem of high computational demands in transformer training by introducing EcoSpa, a structured sparse training method that jointly sparsifies coupled weight matrix pairs to preserve interaction patterns, resulting in 50% memory reduction and 21% faster training for LLaMA-1B, 2.2x model compression with lower perplexity for GPT-2-Medium, and 1.6x inference speedup.
Transformers have become the backbone of modern AI, yet their high computational demands pose critical system challenges. While sparse training offers efficiency gains, existing methods fail to preserve critical structural relationships between weight matrices that interact multiplicatively in attention and feed-forward layers. This oversight leads to performance degradation at high sparsity levels. We introduce EcoSpa, an efficient structured sparse training method that jointly evaluates and sparsifies coupled weight matrix pairs, preserving their interaction patterns through aligned row/column removal. EcoSpa introduces a new granularity for calibrating structural component importance and performs coupled estimation and sparsification across both pre-training and fine-tuning scenarios. Evaluations demonstrate substantial improvements: EcoSpa enables efficient training of LLaMA-1B with 50\% memory reduction and 21\% faster training, achieves $2.2\times$ model compression on GPT-2-Medium with $2.4$ lower perplexity, and delivers $1.6\times$ inference speedup. The approach uses standard PyTorch operations, requiring no custom hardware or kernels, making efficient transformer training accessible on commodity hardware.