AI evaluation may bias perceptions: The importance of context in interpreting academic writing
For researchers and policymakers measuring AI use in science, this work highlights a critical confound in existing evaluation methods.
The paper shows that ignoring contextual differences across countries and fields biases AI-likeness benchmarks, causing systematic over- and underestimation of AI use in scientific writing. Using country-field-specific benchmarks attenuates these distortions.
This paper examines how estimates of AI use in scientific writing can be biased when evaluation methods ignore contextual differences across countries and fields. Using large-scale data on journal publications from Dimensions, we construct AI-likeness benchmarks based on differences between human-written and LLM-rephrased abstracts. We show that a pooled benchmark may confound pre-existing stylistic variation with AI-generated text, producing substantial distortions across country-field groups even in pre-LLM publications. In contrast, country-field-specific benchmarks attenuate such distortions and provide a more credible baseline for comparison. Applying these methods to publications in 2025 reveals that the pooled benchmark systematically overestimates AI use in certain countries and fields while underestimating it in others. These findings highlight the importance of context-aware measurement for accurate and equitable evaluation of AI use in science.