Making Bias Non-Predictive: Training Robust LLM Judges via Reinforcement Learning
This addresses the issue of unreliable automated LLM judges for users relying on AI assessments, though it is an incremental improvement over existing mitigations.
The paper tackles the problem of cognitive biases in LLM judges by proposing Epistemic Independence Training (EIT), a reinforcement learning framework that makes bias cues non-predictive of reward, resulting in improved accuracy and robustness under adversarial biases while preserving performance when bias aligns with truth, with experiments on Qwen3-4B showing generalization to unseen bias types.
Large language models (LLMs) increasingly serve as automated judges, yet they remain susceptible to cognitive biases -- often altering their reasoning when faced with spurious prompt-level cues such as consensus claims or authority appeals. Existing mitigations via prompting or supervised fine-tuning fail to generalize, as they modify surface behavior without changing the optimization objective that makes bias cues predictive. To address this gap, we propose Epistemic Independence Training (EIT), a reinforcement learning framework grounded in a key principle: to learn independence, bias cues must be made non-predictive of reward. EIT operationalizes this through a balanced conflict strategy where bias signals are equally likely to support correct and incorrect answers, combined with a reward design that penalizes bias-following without rewarding bias agreement. Experiments on Qwen3-4B demonstrate that EIT improves both accuracy and robustness under adversarial biases, while preserving performance when bias aligns with truth. Notably, models trained only on bandwagon bias generalize to unseen bias types such as authority and distraction, indicating that EIT induces transferable epistemic independence rather than bias-specific heuristics. Code and data are available at https://anonymous.4open.science/r/bias-mitigation-with-rl-BC47.