Towards Watermarking of Open-Source LLMs
This work addresses the critical need for durable watermarks in open-source LLMs to prevent misuse, but it is incremental as it lays groundwork without presenting a new solution.
The paper tackles the problem of watermarking open-source large language models (LLMs), which lack existing durable methods, by formulating key requirements and proposing an evaluation setup, showing current methods are not durable against common modifications.
While watermarks for closed LLMs have matured and have been included in large-scale deployments, these methods are not applicable to open-source models, which allow users full control over the decoding process. This setting is understudied yet critical, given the rising performance of open-source models. In this work, we lay the foundation for systematic study of open-source LLM watermarking. For the first time, we explicitly formulate key requirements, including durability against common model modifications such as model merging, quantization, or finetuning, and propose a concrete evaluation setup. Given the prevalence of these modifications, durability is crucial for an open-source watermark to be effective. We survey and evaluate existing methods, showing that they are not durable. We also discuss potential ways to improve their durability and highlight remaining challenges. We hope our work enables future progress on this important problem.