SEQR: Secure and Efficient QR-based LoRA Routing
This work addresses the problem of dynamic LoRA composition for large language models, particularly in privacy-sensitive settings, offering a scalable solution.
The paper tackles the challenge of efficiently selecting the correct LoRA adapter for inputs in secure environments by introducing SEQR, an unsupervised routing algorithm that maximizes activation norms, resulting in improved multi-task performance and efficiency.
Low-Rank Adaptation (LoRA) has become a standard technique for parameter-efficient fine-tuning of large language models, enabling large libraries of LoRAs, each for a specific task or domain. Efficiently selecting the correct LoRA adapter for a given input remains a challenge, particularly in secure environments where supervised training of routers may raise privacy concerns. Motivated by previous approaches, we formalize the goal of unsupervised LoRA routing in terms of activation norm maximization, providing a theoretical framework for analysis. We demonstrate the discriminative power of activation norms and introduce SEQR, an unsupervised LoRA routing algorithm designed to maximize efficiency while providing strict routing guarantees. SEQR provably identifies the norm-maximizing adapter with significantly greater efficiency, making it a highly scalable and effective solution for dynamic LoRA composition. We validate our results through experiments that demonstrate improved multi-task performance and efficiency.