Robust Reward Modeling for Large Language Models via Causal Decomposition
For practitioners aligning LLMs, this provides a robust reward modeling approach that reduces reliance on length and sycophancy artifacts, improving alignment quality.
Reward models for LLMs overfit to spurious cues like response length. The authors propose a causal decomposition method that regularizes reward training using reconstruction error from a latent intent decoder, improving RewardBench accuracy from 0.832 to 0.868 on Gemma-2 models and achieving 0.877 accuracy in selecting shorter, less sycophantic candidates.
Reward models are central to aligning large language models, yet they often overfit to spurious cues such as response length and overly agreeable tone. Most prior work weakens these cues directly by penalizing or controlling specific artifacts, but it does not explicitly encourage the model to ground preferences in the prompt's intent. We learn a decoder that maps a candidate answer to the latent intent embedding of the input. The reconstruction error is used as a signal to regularize the reward model training. We provide theoretical evidence that this signal emphasizes prompt-dependent information while suppressing prompt-independent shortcuts. Across math, helpfulness, and safety benchmarks, the decoder selects shorter and less sycophantic candidates with 0.877 accuracy. Incorporating this signal into RM training in Gemma-2-2B-it and Gemma-2-9B-it increases RewardBench accuracy from 0.832 to 0.868. For Best-of-N selection, our framework increases length-controlled win rates while producing shorter outputs, and remains robust to lengthening and mild off-topic drift in controlled rewrite tests.