Effective LoRA Adapter Routing using Task Representations
This addresses the challenge of efficiently utilizing growing public adapter pools for LLM specialization, offering a scalable solution for researchers and practitioners in natural language processing.
The paper tackles the problem of selecting and composing LoRA adapters for large language models by introducing LORAUTER, a routing framework that uses task representations instead of adapter characteristics, achieving state-of-the-art results with a 101.2% match to Oracle performance and a +5.2 point gain on unseen tasks.
Low-rank adaptation (LoRA) enables parameter efficient specialization of large language models (LLMs) through modular adapters, resulting in rapidly growing public adapter pools spanning diverse tasks. Effectively using these adapters requires routing: selecting and composing the appropriate adapters for a query. We introduce LORAUTER, a novel routing framework that selects and composes LoRA adapters using task representations rather than adapter characteristics. Unlike existing approaches that map queries directly to adapters, LORAUTER routes queries via task embeddings derived from small validation sets and does not require adapter training data. By operating at the task level, LORAUTER achieves efficient routing that scales with the number of tasks rather than the number of adapters. Experiments across multiple tasks show that LORAUTER consistently outperforms baseline routing approaches, matching Oracle performance (101.2%) when task-aligned adapters exist and achieving state-of-the-art results on unseen tasks (+5.2 points). We further demonstrate the robustness of LORAUTER to very large, noisy adapter pools by scaling it to over 1500 adapters.