The Real, the Better: Aligning Large Language Models with Online Human Behaviors
This addresses the challenge of adapting LLMs to dynamic online human preferences, though it appears incremental as it builds on existing alignment methods.
The paper tackles the problem of aligning large language models with diverse online human preferences by proposing RLHB, a framework that uses real online human behaviors for alignment, achieving effectiveness confirmed through human and automatic evaluations.
Large language model alignment is widely used and studied to avoid LLM producing unhelpful and harmful responses. However, the lengthy training process and predefined preference bias hinder adaptation to online diverse human preferences. To this end, this paper proposes an alignment framework, called Reinforcement Learning with Human Behavior (RLHB), to align LLMs by directly leveraging real online human behaviors. By taking the generative adversarial framework, the generator is trained to respond following expected human behavior; while the discriminator tries to verify whether the triplets of query, response, and human behavior come from real online environments. Behavior modeling in natural-language form and the multi-model joint training mechanism enable an active and sustainable online alignment. Experimental results confirm the effectiveness of our proposed methods by both human and automatic evaluations.