Inference Time Alignment with Reward-Guided Tree Search
This work addresses the challenge of enhancing LLM alignment during inference for users needing more accurate outputs, though it is incremental as it builds on existing tree-search techniques.
The paper tackles the problem of aligning large language models (LLMs) by proposing DARWIN, an inference-time method that uses reward-guided tree search to improve model responses, outperforming other inference-time alignment methods on benchmarks like AlpacaEval 2 and MT-Bench and achieving performance comparable to preference-tuned models.
Inference-time computation methods enhance the performance of Large Language Models (LLMs) by leveraging additional computational resources to achieve superior results. Common techniques, such as Best-of-N sampling, Majority Voting, and variants of tree-search algorithms have proven to be effective in boosting the performance of LLMs. These approaches strategically trade increased computational resources for improved model responses. In this work, we proposed DARWIN, an inference-time alignment method that leverages the guidance of a reward model to achieve alignment through a reward-guided tree search. Empirical evidences indicates that our method outperforms other inference-time alignment methods such as Best-of-N and ARGS on two widely accepted alignment benchmarks AlpacaEval 2 and MT-Bench. Furthermore, we show that our inference-time approach achieves performance comparable to preference-tuned models on both benchmarks, highlighting the effectiveness of trading inference-time compute for enhanced performance during inference. We have released our codes at https://github.com/declare-lab/darwin.