ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment
This work addresses the high cost of collecting explicit feedback for reward modeling in RLHF, offering a cost-effective alternative for aligning language models, though it appears incremental as it builds on existing reward modeling frameworks.
The paper tackles the challenge of learning reward models from implicit human feedback (e.g., clicks and copies) for LLM alignment, proposing ImplicitRM to address issues like lack of negative samples and user preference bias, with experiments showing it learns accurate reward models.
Reward modeling represents a long-standing challenge in reinforcement learning from human feedback (RLHF) for aligning language models. Current reward modeling is heavily contingent upon experimental feedback data with high collection costs. In this work, we study \textit{implicit reward modeling} -- learning reward models from implicit human feedback (e.g., clicks and copies) -- as a cost-effective alternative. We identify two fundamental challenges in implicit reward modeling: (1) Implicit preference data lacks definitive negative samples, which makes standard positive-negative classification methods inapplicable; (2) Implicit preference data suffers from user preference bias, where different responses have different propensities to elicit user feedback actions, which exacerbates the difficulty of distinguishing definitive negative samples. To address these challenges, we propose ImplicitRM, which aims to learn unbiased reward models from implicit preference data. ImplicitRM stratifies training samples into four latent groups via a stratification model. Building on this, it derives a learning objective through likelihood maximization, which we prove is theoretically unbiased, effectively resolving both challenges. Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets. Code is available on our project website.