Guiding LLM Decision-Making with Fairness Reward Models
This addresses fairness issues in high-stakes LLM decisions (e.g., bail, loans) with a transferable method, though it is incremental as it builds on reward modeling techniques.
The paper tackles the problem of unfair bias amplification in LLM decision-making by proposing a Fairness Reward Model that scores reasoning fairness, enabling down-weighting of biased trajectories. The approach improves fairness while matching or surpassing baseline accuracy on tasks like recidivism prediction and social media moderation.
Large language models are increasingly used to support high-stakes decisions, potentially influencing who is granted bail or receives a loan. Naive chain-of-thought sampling can improve average decision accuracy, but has also been shown to amplify unfair bias. To address this challenge and enable the trustworthy use of reasoning models in high-stakes decision-making, we propose a framework for training a generalizable Fairness Reward Model (FRM). Our model assigns a fairness score to LLM reasoning, enabling the system to down-weight biased trajectories and favor equitable ones when aggregating decisions across reasoning chains. We show that a single Fairness Reward Model, trained on weakly supervised, LLM-annotated examples of biased versus unbiased reasoning, transfers across tasks, domains, and model families without additional fine-tuning. Applied to real-world decision-making tasks including recidivism prediction and social media moderation, we show that our approach consistently improves fairness while matching, or even surpassing, baseline accuracy.