MoLoRA: Composable Specialization via Per-Token Adapter Routing
This addresses the need for modular expertise in AI inference, enabling efficient combination of specialized adapters without retraining, though it is incremental in improving adapter routing methods.
The paper tackles the problem of multi-adapter serving systems that route entire sequences to a single adapter, which fails in multimodal generation and mixed-capability requests, by introducing per-token routing that enables composable specialization, allowing Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7x smaller.
Multi-adapter serving systems route entire sequences to a single adapter, forcing a choice when requests span multiple domains. This assumption fails in two important settings: (1) multimodal generation, where text and image tokens require different adapters within the same sequence, and (2) mixed-capability requests like "write code to solve this equation," which need expertise from multiple specialized adapters. We introduce per-token routing, which routes individual tokens to adapters based on either vocabulary structure (for multimodal models) or learned gating (for semantic specialization). Per-token routing is provably optimal, achieving work N for N tokens versus K \cdot N for per-sequence routing with K adapter types. Our key contribution is MoLoRA (Mixture of LoRA), which enables composable specialization: load multiple domain-specific adapters and let a learned router select the appropriate adapter per-token. We demonstrate that specialization dramatically beats scale: MoLoRA enables Qwen3-1.7B to exceed Qwen3-8B across four reasoning benchmarks while being 4.7x smaller. This enables modular expertise at inference time: train focused LoRAs independently, combine them without retraining, and add new capabilities by simply loading new adapters.