Watermarking Pre-trained Language Models with Backdooring
This addresses the issue of intellectual property protection for developers of PLMs, though it is an incremental improvement in watermarking techniques.
The paper tackles the problem of claiming ownership and protecting intellectual property for pre-trained language models (PLMs) by watermarking them with backdoors triggered by specific inputs, showing that these watermarks remain robust with a high success rate even after fine-tuning on multiple downstream tasks.
Large pre-trained language models (PLMs) have proven to be a crucial component of modern natural language processing systems. PLMs typically need to be fine-tuned on task-specific downstream datasets, which makes it hard to claim the ownership of PLMs and protect the developer's intellectual property due to the catastrophic forgetting phenomenon. We show that PLMs can be watermarked with a multi-task learning framework by embedding backdoors triggered by specific inputs defined by the owners, and those watermarks are hard to remove even though the watermarked PLMs are fine-tuned on multiple downstream tasks. In addition to using some rare words as triggers, we also show that the combination of common words can be used as backdoor triggers to avoid them being easily detected. Extensive experiments on multiple datasets demonstrate that the embedded watermarks can be robustly extracted with a high success rate and less influenced by the follow-up fine-tuning.