Preference-Guided Reflective Sampling for Aligning Language Models
This work addresses the challenge of efficient data sampling for aligning LLMs to diverse user preferences, representing an incremental improvement over existing methods.
The paper tackles the problem of aligning large language models to human preferences by proposing Preference-Guided Reflective Sampling (PRS), a tree-based sampling method that uses adaptive self-refinement and natural language preferences to generate higher-quality responses, resulting in significantly higher rewards on benchmarks like AlpacaEval and Arena-Hard compared to random sampling.
Iterative data generation and model re-training can effectively align large language models(LLMs) to human preferences. The process of data sampling is crucial, as it significantly influences the success of policy improvement. Repeated random sampling is a widely used method that independently queries the model multiple times to generate outputs. In this work, we propose a more effective sampling method, named Preference-Guided Reflective Sampling (PRS). Unlike random sampling, PRS employs a tree-based generation framework to enable more efficient sampling. It leverages adaptive self-refinement techniques to better explore the sampling space. By specifying user preferences in natural language, PRS can further optimize response generation according to these preferences. As a result, PRS can align models to diverse user preferences. Our experiments demonstrate that PRS generates higher-quality responses with significantly higher rewards. On AlpacaEval and Arena-Hard, PRS substantially outperforms repeated random sampling in best-of-$N$ sampling. Moreover, PRS shows strong performance when applied in iterative offline RL training.