MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
This work addresses efficiency problems for practitioners fine-tuning LLMs in multi-tenant environments, though it is incremental as it builds on existing LoRA and MOE methods.
The paper tackles the latency issue in multi-tenant fine-tuning of large language models with LoRA and MOE variants by proposing MiLoRA, which uses a prompt-aware routing mechanism to reduce latency while maintaining performance, achieving significant latency reductions and outperforming baselines on reasoning tasks and benchmarks.
Low-rank adaptation (LoRA) and its mixture-of-experts (MOE) variants are highly effective parameter-efficient fine-tuning (PEFT) methods. However, they introduce significant latency in multi-tenant settings due to the LoRA modules and MOE routers added to multiple linear modules in the Transformer layer. To address this issue, we propose Mixture of Low-Rank Adaptation (MiLoRA), a novel and efficient LoRA variant. MiLoRA differs from previous MOE-style LoRA methods by considering each LoRA module as an expert and employing a prompt-aware routing mechanism. This mechanism calculates expert routing results once before generating the first new token and reuses these results for subsequent tokens, reducing latency. Extensive experiments and analysis on commonsense reasoning tasks, math reasoning tasks, and widely used LLM evaluation benchmarks demonstrate that MiLoRA consistently outperforms strong PEFT baselines with comparable tunable parameter budgets. Additionally, MiLoRA significantly reduces latency in multi-tenant settings compared to previous LoRA-based methods.