Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment
This work addresses the problem of simplifying alignment for LLM developers by offering an alternative to RLHF, though it is incremental as it builds on existing SFT methods.
The paper tackles the limitation of supervised fine-tuning for aligning large language models with human preferences by proposing FIGA, which incorporates fine-grained quality signals from contrasting good and bad responses, resulting in demonstrated effectiveness against competitive baselines.
Alignment with human preference is a desired property of large language models (LLMs). Currently, the main alignment approach is based on reinforcement learning from human feedback (RLHF). Despite the effectiveness of RLHF, it is intricate to implement and train, thus recent studies explore how to develop alternative alignment approaches based on supervised fine-tuning (SFT). A major limitation of SFT is that it essentially does imitation learning, which cannot fully understand what are the expected behaviors. To address this issue, we propose an improved alignment approach named FIGA. Different from prior methods, we incorporate fine-grained (i.e., token or phrase level) quality signals that are derived by contrasting good and bad responses. Our approach has made two major contributions. Firstly, we curate a refined alignment dataset that pairs initial responses and the corresponding revised ones. Secondly, we devise a new loss function can leverage fine-grained quality signals to instruct the learning of LLMs for alignment. Extensive experiments have demonstrated the effectiveness of our approaches by comparing a number of competitive baselines.