In-Context Alignment: Chat with Vanilla Language Models Before Fine-Tuning
This addresses the challenge of aligning language models for chat-style instructions efficiently, though it appears incremental as it builds on existing in-context learning methods.
The paper tackles the problem of aligning language models with human preferences without fine-tuning by using in-context learning with retrieved demonstration examples, resulting in a 7x increase in win-rate compared to direct prompting and making the vanilla model competitive with fine-tuned baselines.
In this note, we explore inference-time alignment through in-context learning. We consider a vanilla pretrained language model Llama-2 before any fine-tuning and retrieve an average of 9 demonstration alignment examples when the model is prompted to follow chat-style instructions. Compared to direct prompting, the in-context alignment without changing model weights leads to a 7x increase in win-rate w.r.t. the text-davinci-003 model from OpenAI, making the vanilla language model comparable to strong baselines with alignment fine-tuning.