SocioEval: A Template-Based Framework for Evaluating Socioeconomic Status Bias in Foundation Models
This addresses socioeconomic status bias in LLMs for responsible AI deployment, which is an incremental contribution as it extends bias assessment to an underexplored attribute.
The authors tackled the problem of evaluating socioeconomic status bias in foundation models by introducing SocioEval, a template-based framework, and found substantial variation in bias rates (0.42%-33.75%) across 13 frontier LLMs, with lifestyle judgments showing 10x higher bias than education-related decisions.
As Large Language Models (LLMs) increasingly power decision-making systems across critical domains, understanding and mitigating their biases becomes essential for responsible AI deployment. Although bias assessment frameworks have proliferated for attributes such as race and gender, socioeconomic status bias remains significantly underexplored despite its widespread implications in the real world. We introduce SocioEval, a template-based framework for systematically evaluating socioeconomic bias in foundation models through decision-making tasks. Our hierarchical framework encompasses 8 themes and 18 topics, generating 240 prompts across 6 class-pair combinations. We evaluated 13 frontier LLMs on 3,120 responses using a rigorous three-stage annotation protocol, revealing substantial variation in bias rates (0.42\%-33.75\%). Our findings demonstrate that bias manifests differently across themes lifestyle judgments show 10$\times$ higher bias than education-related decisions and that deployment safeguards effectively prevent explicit discrimination but show brittleness to domain-specific stereotypes. SocioEval provides a scalable, extensible foundation for auditing class-based bias in language models.