Black-box Detection of LLM-generated Text Using Generalized Jensen-Shannon Divergence
This addresses the problem of detecting AI-generated text for applications like content moderation, but it is incremental as it builds on existing detection methods with a novel scoring approach.
The paper tackled the problem of black-box detection of machine-generated text under practical constraints like model mismatch and high computational cost, proposing SurpMark, a reference-based detector that uses generalized Jensen-Shannon divergence on token surprisal dynamics, which consistently matched or surpassed baselines across multiple datasets and scenarios.
We study black-box detection of machine-generated text under practical constraints: the scoring model (proxy LM) may mismatch the unknown source model, and per-input contrastive generation is costly. We propose SurpMark, a reference-based detector that summarizes a passage by the dynamics of its token surprisals. SurpMark quantizes surprisals into interpretable states, estimates a state-transition matrix for the test text, and scores it via a generalized Jensen-Shannon (GJS) gap between the test transitions and two fixed references (human vs. machine) built once from historical corpora. We prove a principled discretization criterion and establish the asymptotic normality of the decision statistic. Empirically, across multiple datasets, source models, and scenarios, SurpMark consistently matches or surpasses baselines; our experiments corroborate the statistic's asymptotic normality, and ablations validate the effectiveness of the proposed discretization.