Smaller Language Models are Better Black-box Machine-Generated Text Detectors
This addresses the challenge of identifying AI-generated content to combat misinformation, though it is incremental as it builds on existing detection methods.
The paper tackles the problem of detecting machine-generated text by showing that smaller language models, such as OPT-125M, are more effective black-box detectors than larger ones, achieving an AUC of 0.81 against ChatGPT compared to GPTJ-6B's 0.45.
With the advent of fluent generative language models that can produce convincing utterances very similar to those written by humans, distinguishing whether a piece of text is machine-generated or human-written becomes more challenging and more important, as such models could be used to spread misinformation, fake news, fake reviews and to mimic certain authors and figures. To this end, there have been a slew of methods proposed to detect machine-generated text. Most of these methods need access to the logits of the target model or need the ability to sample from the target. One such black-box detection method relies on the observation that generated text is locally optimal under the likelihood function of the generator, while human-written text is not. We find that overall, smaller and partially-trained models are better universal text detectors: they can more precisely detect text generated from both small and larger models. Interestingly, we find that whether the detector and generator were trained on the same data is not critically important to the detection success. For instance the OPT-125M model has an AUC of 0.81 in detecting ChatGPT generations, whereas a larger model from the GPT family, GPTJ-6B, has AUC of 0.45.