ReFT: Representation Finetuning for Language Models
This work addresses the need for more efficient and effective fine-tuning methods for large language models, offering a novel approach that could benefit researchers and practitioners in NLP, though it appears incremental as it builds on existing PEFT concepts.
The authors tackled the problem of parameter-efficient fine-tuning (PEFT) for large language models by proposing Representation Finetuning (ReFT) methods, which edit hidden representations instead of weights, resulting in interventions that are 15x to 65x more parameter-efficient than LoRA and outperforming state-of-the-art PEFTs across multiple reasoning and GLUE tasks.
Parameter-efficient finetuning (PEFT) methods seek to adapt large neural models via updates to a small number of weights. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative. We pursue this hypothesis by developing a family of Representation Finetuning (ReFT) methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT), and we identify an ablation of this method that trades some performance for increased efficiency. Both are drop-in replacements for existing PEFTs and learn interventions that are 15x--65x more parameter-efficient than LoRA. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, instruction-tuning, and GLUE. In all these evaluations, our ReFTs deliver the best balance of efficiency and performance, and almost always outperform state-of-the-art PEFTs. We release a generic ReFT training library publicly at https://github.com/stanfordnlp/pyreft.