HALF: Harm-Aware LLM Fairness Evaluation Aligned with Deployment
This addresses fairness evaluation for LLM deployment in high-impact domains like healthcare and education, but it is incremental as it builds on existing evaluation methods by adding harm severity weighting.
The paper tackles the problem of evaluating fairness in large language models (LLMs) for real-world deployment by introducing HALF, a harm-aware framework that assesses bias across nine application domains weighted by harm severity, showing that LLMs are not consistently fair and that model size or performance does not guarantee fairness.
Large language models (LLMs) are increasingly deployed across high-impact domains, from clinical decision support and legal analysis to hiring and education, making fairness and bias evaluation before deployment critical. However, existing evaluations lack grounding in real-world scenarios and do not account for differences in harm severity, e.g., a biased decision in surgery should not be weighed the same as a stylistic bias in text summarization. To address this gap, we introduce HALF (Harm-Aware LLM Fairness), a deployment-aligned framework that assesses model bias in realistic applications and weighs the outcomes by harm severity. HALF organizes nine application domains into three tiers (Severe, Moderate, Mild) using a five-stage pipeline. Our evaluation results across eight LLMs show that (1) LLMs are not consistently fair across domains, (2) model size or performance do not guarantee fairness, and (3) reasoning models perform better in medical decision support but worse in education. We conclude that HALF exposes a clear gap between previous benchmarking success and deployment readiness.