Detecting Post-generation Edits to Watermarked LLM Outputs via Combinatorial Watermarking
This addresses a critical issue for proprietary language models in real-world scenarios where AI-generated content may be modified, though it is incremental as it builds on existing watermarking techniques.
The paper tackles the problem of detecting and localizing post-generation edits to watermarked LLM outputs, such as human revisions or spoofing attacks, by proposing a combinatorial pattern-based watermarking framework that achieves strong empirical performance in edit localization.
Watermarking has become a key technique for proprietary language models, enabling the distinction between AI-generated and human-written text. However, in many real-world scenarios, LLM-generated content may undergo post-generation edits, such as human revisions or even spoofing attacks, making it critical to detect and localize such modifications. In this work, we introduce a new task: detecting post-generation edits locally made to watermarked LLM outputs. To this end, we propose a combinatorial pattern-based watermarking framework, which partitions the vocabulary into disjoint subsets and embeds the watermark by enforcing a deterministic combinatorial pattern over these subsets during generation. We accompany the combinatorial watermark with a global statistic that can be used to detect the watermark. Furthermore, we design lightweight local statistics to flag and localize potential edits. We introduce two task-specific evaluation metrics, Type-I error rate and detection accuracy, and evaluate our method on open-source LLMs across a variety of editing scenarios, demonstrating strong empirical performance in edit localization.