REAR: Test-time Preference Realignment through Reward Decomposition
This work addresses the challenge of adapting LLMs to diverse user preferences without costly retraining, offering a training-free test-time solution for preference alignment.
The paper introduces REAR, a test-time framework that realigns LLMs with diverse user preferences by decomposing the reward function into question- and preference-related components, enabling efficient integration with test-time scaling algorithms. Experiments show REAR outperforms test-time baselines on preference alignment tasks and generalizes to math and visual tasks.
Aligning large language models (LLMs) with diverse user preferences is a critical yet challenging task. While post-training methods can adapt models to specific needs, they often require costly data curation and additional training. Test-time scaling (TTS) presents an efficient, training-free alternative, but its application has been largely limited to verifiable domains like mathematics and coding, where response correctness is easily judged. To extend TTS to preference alignment, we introduce a novel framework that models the task as a realignment problem, since the base model often fails to sufficiently align with the stated preference. Our key insight is to decompose the underlying reward function into two components: one related to the question and the other to preference information. This allows us to derive a REAlignment Reward (REAR) that selectively rescales the proportions of these two reward terms. We then show that REAR can be formulated as a linear combination of token-level policy log-probabilities, making it computationally efficient and easy to integrate with various TTS algorithms such as best-of-$N$ sampling and tree search. Experiments show that compared to other test-time baselines, REAR not only enables scalable test-time realignment for preference alignment tasks under diverse user requirements, but also generalizes to mathematical and visual tasks under appropriate preference settings.