A Comprehensive Survey of LLM Alignment Techniques: RLHF, RLAIF, PPO, DPO and More
This is an incremental work that addresses a gap for researchers and practitioners by organizing existing alignment methods.
The paper tackles the lack of a comprehensive survey on LLM alignment techniques by categorizing and detailing methods like RLHF, RLAIF, PPO, and DPO, providing readers with a thorough understanding of the field's current state.
With advancements in self-supervised learning, the availability of trillions tokens in a pre-training corpus, instruction fine-tuning, and the development of large Transformers with billions of parameters, large language models (LLMs) are now capable of generating factual and coherent responses to human queries. However, the mixed quality of training data can lead to the generation of undesired responses, presenting a significant challenge. Over the past two years, various methods have been proposed from different perspectives to enhance LLMs, particularly in aligning them with human expectation. Despite these efforts, there has not been a comprehensive survey paper that categorizes and details these approaches. In this work, we aim to address this gap by categorizing these papers into distinct topics and providing detailed explanations of each alignment method, thereby helping readers gain a thorough understanding of the current state of the field.