Training Neural Networks from Scratch with Parallel Low-Rank Adapters
This work addresses scalability limitations in deep learning pre-training for researchers and practitioners, but it is incremental as it builds on existing LoRA methods.
The paper tackled the problem of extending low-rank adaptation (LoRA) to model pre-training by introducing LoRA-the-Explorer (LTE), a bi-level optimization algorithm for parallel training of low-rank heads, and demonstrated that LTE is competitive with standard pre-training on vision transformers across various datasets.
The scalability of deep learning models is fundamentally limited by computing resources, memory, and communication. Although methods like low-rank adaptation (LoRA) have reduced the cost of model finetuning, its application in model pre-training remains largely unexplored. This paper explores extending LoRA to model pre-training, identifying the inherent constraints and limitations of standard LoRA in this context. We introduce LoRA-the-Explorer (LTE), a novel bi-level optimization algorithm designed to enable parallel training of multiple low-rank heads across computing nodes, thereby reducing the need for frequent synchronization. Our approach includes extensive experimentation on vision transformers using various vision datasets, demonstrating that LTE is competitive with standard pre-training.