NeuroLoRA: Context-Aware Neuromodulation for Parameter-Efficient Multi-Task Adaptation
This work addresses the need for more adaptive and efficient multi-task adaptation in AI, though it is incremental as it builds on existing LoRA and MoE methods.
The paper tackled the problem of static routing in parameter-efficient fine-tuning for large language models by proposing NeuroLoRA, a context-aware neuromodulation framework that outperformed FlyLoRA and other baselines on tasks like MMLU, GSM8K, and ScienceQA while maintaining parameter efficiency.
Parameter-Efficient Fine-Tuning (PEFT) techniques, particularly Low-Rank Adaptation (LoRA), have become essential for adapting Large Language Models (LLMs) to downstream tasks. While the recent FlyLoRA framework successfully leverages bio-inspired sparse random projections to mitigate parameter interference, it relies on a static, magnitude-based routing mechanism that is agnostic to input context. In this paper, we propose NeuroLoRA, a novel Mixture-of-Experts (MoE) based LoRA framework inspired by biological neuromodulation -- the dynamic regulation of neuronal excitability based on context. NeuroLoRA retains the computational efficiency of frozen random projections while introducing a lightweight, learnable neuromodulation gate that contextually rescales the projection space prior to expert selection. We further propose a Contrastive Orthogonality Loss to explicitly enforce separation between expert subspaces, enhancing both task decoupling and continual learning capacity. Extensive experiments on MMLU, GSM8K, and ScienceQA demonstrate that NeuroLoRA consistently outperforms FlyLoRA and other strong baselines across single-task adaptation, multi-task model merging, and sequential continual learning scenarios, while maintaining comparable parameter efficiency.