Race, Ethnicity and Their Implication on Bias in Large Language Models
This is an incremental study addressing bias in LLMs for high-stakes applications like healthcare, but it offers mechanistic insights rather than just documenting disparities.
The study investigated how race and ethnicity are represented in large language models, finding that demographic information is distributed across internal units with cross-model variation and that interventions to reduce bias have limited effectiveness, leaving substantial residual effects.
Large language models (LLMs) increasingly operate in high-stakes settings including healthcare and medicine, where demographic attributes such as race and ethnicity may be explicitly stated or implicitly inferred from text. However, existing studies primarily document outcome-level disparities, offering limited insight into internal mechanisms underlying these effects. We present a mechanistic study of how race and ethnicity are represented and operationalized within LLMs. Using two publicly available datasets spanning toxicity-related generation and clinical narrative understanding tasks, we analyze three open-source models with a reproducible interpretability pipeline combining probing, neuron-level attribution, and targeted intervention. We find that demographic information is distributed across internal units with substantial cross-model variation. Although some units encode sensitive or stereotype-related associations from pretraining, identical demographic cues can induce qualitatively different behaviors. Interventions suppressing such neurons reduce bias but leave substantial residual effects, suggesting behavioral rather than representational change and motivating more systematic mitigation.