Cross-Attention Watermarking of Large Language Models
This addresses the need for watermarking in language models to detect generated text, but it is incremental as it builds on existing watermarking techniques with a novel method.
The paper tackles the problem of imperceptibly watermarking language model outputs to preserve readability and meaning, using a cross-attention mechanism to embed watermarks during inference with minimal performance impact, and explores tradeoffs between robustness and text quality.
A new approach to linguistic watermarking of language models is presented in which information is imperceptibly inserted into the output text while preserving its readability and original meaning. A cross-attention mechanism is used to embed watermarks in the text during inference. Two methods using cross-attention are presented that minimize the effect of watermarking on the performance of a pretrained model. Exploration of different training strategies for optimizing the watermarking and of the challenges and implications of applying this approach in real-world scenarios clarified the tradeoff between watermark robustness and text quality. Watermark selection substantially affects the generated output for high entropy sentences. This proactive watermarking approach has potential application in future model development.