Robust Data Watermarking in Language Models by Injecting Fictitious Knowledge
This addresses copyright protection for data owners in AI by providing a more robust watermarking method against preprocessing and verification challenges, though it is incremental as it builds on existing techniques.
The paper tackles the problem of data watermarking in language models by proposing an approach that injects plausible fictitious knowledge into training data, demonstrating effective memorization by LLMs with improvements from increased watermark density, length, and diversity, and showing robustness across training stages and API-only access.
Data watermarking in language models injects traceable signals, such as specific token sequences or stylistic patterns, into copyrighted text, allowing copyright holders to track and verify training data ownership. Previous data watermarking techniques primarily focus on effective memorization during pretraining, while overlooking challenges that arise in other stages of the LLM lifecycle, such as the risk of watermark filtering during data preprocessing and verification difficulties due to API-only access. To address these challenges, we propose a novel data watermarking approach that injects plausible yet fictitious knowledge into training data using generated passages describing a fictitious entity and its associated attributes. Our watermarks are designed to be memorized by the LLM through seamlessly integrating in its training data, making them harder to detect lexically during preprocessing. We demonstrate that our watermarks can be effectively memorized by LLMs, and that increasing our watermarks' density, length, and diversity of attributes strengthens their memorization. We further show that our watermarks remain effective after continual pretraining and supervised finetuning. Finally, we show that our data watermarks can be evaluated even under API-only access via question answering.