Run LoRA Run: Faster and Lighter LoRA Implementations
This work addresses efficiency issues for practitioners using LoRA in large language models, but it is incremental as it focuses on implementation optimizations rather than new algorithmic insights.
The paper tackled the problem of slow training and fine-tuning with LoRA (low-rank adapters) by introducing the RunLoRA framework, which optimizes computation graphs based on FLOPs and time estimates, resulting in up to 28% speedup on language modeling networks without accuracy loss.
LoRA is a technique that reduces the number of trainable parameters in a neural network by introducing low-rank adapters to linear layers. This technique is used both for fine-tuning and full training of large language models. This paper presents the RunLoRA framework for efficient implementations of LoRA that significantly improves the speed of neural network training and fine-tuning using low-rank adapters. The proposed implementation optimizes the computation of LoRA operations based on dimensions of corresponding linear layer, layer input dimensions and lora rank by choosing best forward and backward computation graph based on FLOPs and time estimations, resulting in faster training without sacrificing accuracy. The experimental results show up to 28\% speedup on language modeling networks.