ORAL: Prompting Your Large-Scale LoRAs via Conditional Recurrent Diffusion
This addresses the challenge of scalable and controllable parameter generation for evolving LLMs, offering an incremental improvement over existing methods.
The paper tackles the problem of efficiently adapting large language models (LLMs) via Low-Rank Adaptation (LoRA) without costly retraining, by introducing ORAL, a conditional recurrent diffusion framework that generates task-specific LoRA parameters, achieving comparable or superior performance to trained counterparts across multiple tasks and models.
Parameter generation has emerged as a novel paradigm for neural network development, offering an alternative to traditional neural network training by synthesizing high-quality model weights directly. In the context of Low-Rank Adaptation (LoRA) for evolving ($\textit{i.e.}$, constantly updated) large language models (LLMs), this approach promises efficient adaptation without costly retraining. However, existing methods face critical limitations in simultaneously achieving scalability and controllability. In this paper, we introduce $\texttt{ORAL}$, a novel $\textbf{conditional recurrent diffusion}$ framework that addresses these challenges. $\texttt{ORAL}$ incorporates a novel conditioning mechanism that integrates model architecture and textual task specifications, enabling the generation of task-specific LoRA parameters that can seamlessly transfer across evolving foundation models. Our approach successfully scales to billions-of-parameter LLMs and maintains controllability. Through extensive experiments across seven language tasks, four vision tasks, and three multimodal tasks using five pre-trained LLMs, we demonstrate that $\texttt{ORAL}$ generates high-quality LoRA parameters that achieve comparable or superior performance to vanilla trained counterparts.