In-Context Reward Adaptation for Robust Preference Modeling
For researchers and practitioners in human-AI alignment, this work addresses the critical limitation of static reward models by enabling adaptation to heterogeneous and unseen human preferences, though the novelty is incremental as it builds on existing transformer in-context learning.
The paper tackles the problem of static reward models in RLHF failing to generalize to diverse and unseen human preferences. It proposes In-Context Reward Adaptation, a transformer-based framework that adapts to new preference domains using a few demonstrations, achieving robust preference modeling without retraining.
Reinforcement Learning from Human Feedback (RLHF) typically relies on static reward models to align Large Language Models with human preferences. However, human values are inherently diverse and heterogeneous, and a single reward model often lacks the robustness required to generalize to unseen preference domains. While existing multi-reward frameworks attempt to address this, they are often restricted to a fixed set of known domains and fail to adapt to unseen human distributions without costly retraining. In this work, we propose In-Context Reward Adaptation, a transformer-based framework designed to model diverse and unseen human preferences on the fly. By leveraging the in-context learning capabilities of transformers, our approach adaptively infers the underlying reward structure from a small set of preference demonstrations. We demonstrate that while a standard transformer architecture is insufficient for this task by characterizing an asymptotic bias to the ground-truth, incorporating human response time as an auxiliary input signal enables the model to successfully adapt to preferences from previously unseen domains. Our findings show that this approach provides a more robust foundation for preference modeling, allowing for the representation of heterogeneous rewards and preference distribution shift, and offering a scalable path toward more flexible human-AI alignment.