Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights
This provides a rapid, zero-shot method for customizing large language models across tasks, reducing computational costs and time for users in AI and NLP, though it builds on existing PEFT techniques.
The paper tackles the problem of requiring separate optimization runs for each downstream dataset in Parameter-Efficient Fine-Tuning (PEFT) by introducing Drag-and-Drop LLMs (DnD), a prompt-conditioned parameter generator that maps task prompts to LoRA weight updates, resulting in up to 12,000× lower overhead than full fine-tuning and average gains up to 30% in performance over training LoRAs on unseen benchmarks.
Modern Parameter-Efficient Fine-Tuning (PEFT) methods such as low-rank adaptation (LoRA) reduce the cost of customizing large language models (LLMs), yet still require a separate optimization run for every downstream dataset. We introduce \textbf{Drag-and-Drop LLMs (\textit{DnD})}, a prompt-conditioned parameter generator that eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates. A lightweight text encoder distills each prompt batch into condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices. Once trained in a diverse collection of prompt-checkpoint pairs, DnD produces task-specific parameters in seconds, yielding i) up to \textbf{12,000$\times$} lower overhead than full fine-tuning, ii) average gains up to \textbf{30\%} in performance over the strongest training LoRAs on unseen common-sense reasoning, math, coding, and multimodal benchmarks, and iii) robust cross-domain generalization despite never seeing the target data or labels. Our results demonstrate that prompt-conditioned parameter generation is a viable alternative to gradient-based adaptation for rapidly specializing LLMs. Our project is available at \href{https://jerryliang24.github.io/DnD}{https://jerryliang24.github.io/DnD}.