A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
This work addresses the problem of methodological fragmentation for researchers and practitioners using LoRA, offering a systematic framework and tools, but it is incremental as it consolidates existing variants rather than introducing new methods.
This paper tackles the fragmentation in LoRA variants by providing a unified study with taxonomy, theory, codebase, and empirical evaluation, finding that LoRA is sensitive to learning rate and can match or surpass variant performance with proper hyperparameters.
Low-Rank Adaptation (LoRA) is a fundamental parameter-efficient fine-tuning method that balances efficiency and performance in large-scale neural networks. However, the proliferation of LoRA variants has led to fragmentation in methodology, theory, code, and evaluation. To this end, this work presents the first unified study of LoRA variants, offering a systematic taxonomy, unified theoretical review, structured codebase, and standardized empirical assessment. First, we categorize LoRA variants along four principal axes: rank, optimization dynamics, initialization, and integration with Mixture-of-Experts. Then, we review their relationships and evolution within a common theoretical framework focused on low-rank update dynamics. Further, we introduce LoRAFactory, a modular codebase that implements variants through a unified interface, supporting plug-and-play experimentation and fine-grained analysis. Last, using this codebase, we conduct a large-scale evaluation across natural language generation, natural language understanding, and image classification tasks, systematically exploring key hyperparameters. Our results uncover several findings, notably: LoRA and its variants exhibit pronounced sensitivity to the choices of learning rate compared to other hyperparameters; moreover, with proper hyperparameter configurations, LoRA consistently matches or surpasses the performance of most of its variants.