NEFTune: Noisy Embeddings Improve Instruction Finetuning
This addresses a key challenge in instruction finetuning for language models, offering a simple augmentation method that enhances performance across various datasets and models, including those refined with RLHF, though it is incremental as it builds on existing finetuning techniques.
The paper tackled the problem of improving language model finetuning by adding noise to embedding vectors during training, resulting in dramatic performance gains, such as increasing AlpacaEval scores from 29.79% to 64.69% for LLaMA-2-7B and achieving 8-10% improvements on other datasets.
We show that language model finetuning can be improved, sometimes dramatically, with a simple augmentation. NEFTune adds noise to the embedding vectors during training. Standard finetuning of LLaMA-2-7B using Alpaca achieves 29.79% on AlpacaEval, which rises to 64.69% using noisy embeddings. NEFTune also improves over strong baselines on modern instruction datasets. Models trained with Evol-Instruct see a 10% improvement, with ShareGPT an 8% improvement, and with OpenPlatypus an 8% improvement. Even powerful models further refined with RLHF such as LLaMA-2-Chat benefit from additional training with NEFTune.