Functional Invariants to Watermark Large Transformers
This provides a practical, low-cost solution for protecting the integrity and ownership of large AI models, though it is incremental as it builds on existing watermarking concepts.
The paper tackles the problem of computationally expensive watermarking for large transformer models by proposing a method that uses functional invariants, such as dimension permutations, to embed watermarks without altering model outputs, achieving robustness against transformations like fine-tuning and pruning.
The rapid growth of transformer-based models increases the concerns about their integrity and ownership insurance. Watermarking addresses this issue by embedding a unique identifier into the model, while preserving its performance. However, most existing approaches require to optimize the weights to imprint the watermark signal, which is not suitable at scale due to the computational cost. This paper explores watermarks with virtually no computational cost, applicable to a non-blind white-box setting (assuming access to both the original and watermarked networks). They generate functionally equivalent copies by leveraging the models' invariance, via operations like dimension permutations or scaling/unscaling. This enables to watermark models without any change in their outputs and remains stealthy. Experiments demonstrate the effectiveness of the approach and its robustness against various model transformations (fine-tuning, quantization, pruning), making it a practical solution to protect the integrity of large models.