Analyzing Dialectical Biases in LLMs for Knowledge and Reasoning Benchmarks
This addresses bias in LLMs for users of under-represented English dialects, though it is incremental as it builds on known dialect bias issues.
The paper analyzed how converting standard American English questions into non-standard dialectal variants affects LLM performance on multiple-choice QA tasks, finding up to a 20% accuracy reduction. It identified three specific grammar rules (existential 'it', zero copula, and y'all) as explaining most of this degradation across dialects.
Large language models (LLMs) are ubiquitous in modern day natural language processing. However, previous work has shown degraded LLM performance for under-represented English dialects. We analyze the effects of typifying "standard" American English language questions as non-"standard" dialectal variants on multiple choice question answering tasks and find up to a 20% reduction in accuracy. Additionally, we investigate the grammatical basis of under-performance in non-"standard" English questions. We find that individual grammatical rules have varied effects on performance, but some are more consequential than others: three specific grammar rules (existential "it", zero copula, and y'all) can explain the majority of performance degradation observed in multiple dialects. We call for future work to investigate bias mitigation methods focused on individual, high-impact grammatical structures.