Debiasing Reward Models via Causally Motivated Inference-Time Intervention
For practitioners using reward models to align LLMs, this method offers a practical way to mitigate multiple biases at inference time without retraining, though it is incremental as it builds on existing neuron-level intervention techniques.
The paper proposes a causally motivated inference-time intervention that identifies and suppresses neurons correlated with multiple bias attributes in reward models. The method reduces sensitivity to spurious features without performance trade-offs, and when applied to small RMs (2B and 7B), enables LLMs to achieve alignment comparable to a 70B RM on AlpacaEval and MT-Bench.
Reward models (RMs) play a central role in aligning large language models (LLMs) with human preferences. However, RMs are often sensitive to spurious features such as response length. Existing inference-time approaches for mitigating these biases typically focus exclusively on response length, resulting in performance trade-offs. In this paper, we propose causally motivated intervention for mitigating multiple types of biases in RMs at inference time. Our method first identifies neurons whose activations are strongly correlated with predefined bias attributes, and applies neuron-level intervention that suppresses these signals. We evaluate our method on RM benchmarks and observe reductions in sensitivity to spurious features across diverse bias types, without inducing performance trade-offs. Moreover, when used for preference annotation, small RMs (2B and 7B) with our method, which edits less than 2% of all the neurons in RMs, enable LLMs to improve alignment, achieving performance comparable to that of a state-of-the-art 70B RM on AlpacaEval and MT-Bench. Further analysis reveals that bias signals are primarily encoded by neurons in early layers, shedding light on the internal mechanisms of bias exploitation in RMs.