Evaluating Implicit Biases in LLM Reasoning through Logic Grid Puzzles
This addresses the problem of diagnosing and quantifying social biases in LLM deductive reasoning for fairness-critical applications, representing an incremental advance in evaluation methods.
The paper tackled the problem of subtle social biases in LLM reasoning by introducing PRIME, a framework using logic grid puzzles to evaluate implicit biases, finding that models consistently reason more accurately when solutions align with stereotypes, with accuracy differences of up to 15% in some cases.
While recent safety guardrails effectively suppress overtly biased outputs, subtler forms of social bias emerge during complex logical reasoning tasks that evade current evaluation benchmarks. To fill this gap, we introduce a new evaluation framework, PRIME (Puzzle Reasoning for Implicit Biases in Model Evaluation), that uses logic grid puzzles to systematically probe the influence of social stereotypes on logical reasoning and decision making in LLMs. Our use of logic puzzles enables automatic generation and verification, as well as variability in complexity and biased settings. PRIME includes stereotypical, anti-stereotypical, and neutral puzzle variants generated from a shared puzzle structure, allowing for controlled and fine-grained comparisons. We evaluate multiple model families across puzzle sizes and test the effectiveness of prompt-based mitigation strategies. Focusing our experiments on gender stereotypes, our findings highlight that models consistently reason more accurately when solutions align with stereotypical associations. This demonstrates the significance of PRIME for diagnosing and quantifying social biases perpetuated in the deductive reasoning of LLMs, where fairness is critical.