Who Writes What: Unveiling the Impact of Author Roles on AI-generated Text Detection
This work addresses the problem of biased AI text detection that could unfairly penalize specific demographic groups, offering insights for more equitable systems, though it is incremental in focusing on overlooked author characteristics.
The study investigated how sociolinguistic attributes like gender, CEFR proficiency, academic field, and language environment affect AI-generated text detection accuracy, revealing significant biases such as CEFR proficiency and language environment consistently impacting detectors, with gender and academic field showing detector-dependent effects.
The rise of Large Language Models (LLMs) necessitates accurate AI-generated text detection. However, current approaches largely overlook the influence of author characteristics. We investigate how sociolinguistic attributes-gender, CEFR proficiency, academic field, and language environment-impact state-of-the-art AI text detectors. Using the ICNALE corpus of human-authored texts and parallel AI-generated texts from diverse LLMs, we conduct a rigorous evaluation employing multi-factor ANOVA and weighted least squares (WLS). Our results reveal significant biases: CEFR proficiency and language environment consistently affected detector accuracy, while gender and academic field showed detector-dependent effects. These findings highlight the crucial need for socially aware AI text detection to avoid unfairly penalizing specific demographic groups. We offer novel empirical evidence, a robust statistical framework, and actionable insights for developing more equitable and reliable detection systems in real-world, out-of-domain contexts. This work paves the way for future research on bias mitigation, inclusive evaluation benchmarks, and socially responsible LLM detectors.