Disparities in LLM Reasoning Accuracy and Explanations: A Case Study on African American English
This work highlights systematic biases in LLMs that affect users of non-standard dialects like AAE, raising concerns for equitable deployment in multilingual contexts.
The study investigated dialectal disparities in Large Language Models (LLMs) by comparing performance on Standard American English (SAE) and African American English (AAE) prompts, finding that LLMs consistently produced less accurate responses and simpler reasoning for AAE inputs, with disparities most pronounced in social science and humanities domains.
Large Language Models (LLMs) have demonstrated remarkable capabilities in reasoning tasks, leading to their widespread deployment. However, recent studies have highlighted concerning biases in these models, particularly in their handling of dialectal variations like African American English (AAE). In this work, we systematically investigate dialectal disparities in LLM reasoning tasks. We develop an experimental framework comparing LLM performance given Standard American English (SAE) and AAE prompts, combining LLM-based dialect conversion with established linguistic analyses. We find that LLMs consistently produce less accurate responses and simpler reasoning chains and explanations for AAE inputs compared to equivalent SAE questions, with disparities most pronounced in social science and humanities domains. These findings highlight systematic differences in how LLMs process and reason about different language varieties, raising important questions about the development and deployment of these systems in our multilingual and multidialectal world. Our code repository is publicly available at https://github.com/Runtaozhou/dialect_bias_eval.