BAID: A Benchmark for Bias Assessment of AI Detectors
This addresses bias issues in AI detectors for educational and professional users, providing a scalable auditing tool, though it is incremental as it builds on prior isolated findings.
The authors tackled the problem of bias in AI-generated text detectors by proposing BAID, a comprehensive evaluation framework with over 200k samples across 7 sociolinguistic categories, and found consistent disparities in detection performance, such as low recall rates for underrepresented groups.
AI-generated text detectors have recently gained adoption in educational and professional contexts. Prior research has uncovered isolated cases of bias, particularly against English Language Learners (ELLs) however, there is a lack of systematic evaluation of such systems across broader sociolinguistic factors. In this work, we propose BAID, a comprehensive evaluation framework for AI detectors across various types of biases. As a part of the framework, we introduce over 200k samples spanning 7 major categories: demographics, age, educational grade level, dialect, formality, political leaning, and topic. We also generated synthetic versions of each sample with carefully crafted prompts to preserve the original content while reflecting subgroup-specific writing styles. Using this, we evaluate four open-source state-of-the-art AI text detectors and find consistent disparities in detection performance, particularly low recall rates for texts from underrepresented groups. Our contributions provide a scalable, transparent approach for auditing AI detectors and emphasize the need for bias-aware evaluation before these tools are deployed for public use.