Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling
For practitioners of RLHF and LLM alignment, TCRM improves interpretability, process reward modeling, and training efficiency without architectural changes.
Reward models in RLHF discard intermediate token signals. TCRM adds regularization to make token-level outputs represent conditional expectations of final reward, achieving 88.9% middle-token pairwise accuracy, 44.9% F1 on ProcessBench, and 27% GPU memory reduction in PPO.
Reward models in RLHF are trained to score only the final token of a response - a choice that discards rich signal from every intermediate position and produces models whose token-level outputs are noise. We argue this is a missed opportunity: a well-trained reward model's output at any token should represent the conditional expectation of the final reward given the response so far. We introduce Temporally Coherent Reward Modeling (TCRM), which induces this property via two regularization terms on top of the standard Bradley-Terry loss, with minimizers provably equal to conditional expectations. The regularizers correspond to Monte Carlo and TD value-learning objectives, establishing a direct connection to RL value functions. TCRM requires zero changes to architecture, data, or inference, yet unlocks three capabilities from one principle: interpretable token-level reward trajectories (middle-token pairwise accuracy improved from 50% to 88.9%, final-token accuracy preserved); state-of-the-art PRM performance on ProcessBench (44.9% average F1) among models trained only on outcome data; and unified reward/value modeling in PPO, reducing peak GPU memory by 27% and step time by 19% with matching LLM quality.