Different Time, Different Language: Revisiting the Bias Against Non-Native Speakers in GPT Detectors
This addresses concerns about fairness in automated text detection for non-native speakers, though it appears incremental as it revisits prior findings in a specific language.
The study revisited claims of bias against non-native speakers in GPT detectors, specifically in the Czech language, and found no systematic bias, with detectors operating effectively without relying on perplexity.
LLM-based assistants have been widely popularised after the release of ChatGPT. Concerns have been raised about their misuse in academia, given the difficulty of distinguishing between human-written and generated text. To combat this, automated techniques have been developed and shown to be effective, to some extent. However, prior work suggests that these methods often falsely flag essays from non-native speakers as generated, due to their low perplexity extracted from an LLM, which is supposedly a key feature of the detectors. We revisit these statements two years later, specifically in the Czech language setting. We show that the perplexity of texts from non-native speakers of Czech is not lower than that of native speakers. We further examine detectors from three separate families and find no systematic bias against non-native speakers. Finally, we demonstrate that contemporary detectors operate effectively without relying on perplexity.