Invariant-based Robust Weights Watermark for Large Language Models
This addresses the need for protecting IP rights for LLM providers in resource-constrained environments, representing an incremental improvement in watermarking techniques.
The paper tackles the problem of intellectual property theft for large language models deployed on edge devices by introducing a robust watermarking scheme that does not require retraining or fine-tuning, achieving strong robustness against various attacks like fine-tuning and collusion with experimental validation on models such as Llama3, Phi3, and Gemma.
Watermarking technology has gained significant attention due to the increasing importance of intellectual property (IP) rights, particularly with the growing deployment of large language models (LLMs) on billions resource-constrained edge devices. To counter the potential threats of IP theft by malicious users, this paper introduces a robust watermarking scheme without retraining or fine-tuning for transformer models. The scheme generates a unique key for each user and derives a stable watermark value by solving linear constraints constructed from model invariants. Moreover, this technology utilizes noise mechanism to hide watermark locations in multi-user scenarios against collusion attack. This paper evaluates the approach on three popular models (Llama3, Phi3, Gemma), and the experimental results confirm the strong robustness across a range of attack methods (fine-tuning, pruning, quantization, permutation, scaling, reversible matrix and collusion attacks).