Optimization-Inspired Few-Shot Adaptation for Large Language Models
This addresses the challenge of few-shot adaptation for large language models, which is crucial for practical applications with limited data, though it appears incremental as it builds on optimization concepts to improve existing approaches.
The paper tackles the problem of adapting large language models to novel tasks with limited data by proposing Optimization-Inspired Few-Shot Adaptation (OFA), which reinterprets the forward pass as an optimization process and learns preconditioners without extra parameters, resulting in superior performance over existing methods on various few-shot tasks.
Large Language Models (LLMs) have demonstrated remarkable performance in real-world applications. However, adapting LLMs to novel tasks via fine-tuning often requires substantial training data and computational resources that are impractical in few-shot scenarios. Existing approaches, such as in-context learning and Parameter-Efficient Fine-Tuning (PEFT), face key limitations: in-context learning introduces additional inference computational overhead with limited performance gains, while PEFT models are prone to overfitting on the few demonstration examples. In this work, we reinterpret the forward pass of LLMs as an optimization process, a sequence of preconditioned gradient descent steps refining internal representations. Based on this connection, we propose Optimization-Inspired Few-Shot Adaptation (OFA), integrating a parameterization that learns preconditioners without introducing additional trainable parameters, and an objective that improves optimization efficiency by learning preconditioners based on a convergence bound, while simultaneously steering the optimization path toward the flat local minimum. Our method overcomes both issues of ICL-based and PEFT-based methods, and demonstrates superior performance over the existing methods on a variety of few-shot adaptation tasks in experiments.