Ideal Attribution and Faithful Watermarks for Language Models
This provides a foundational framework for researchers and practitioners working on attribution and watermarking in language models, though it is incremental in formalizing existing concepts.
The paper tackles the problem of attribution decisions for language model outputs by introducing ideal attribution mechanisms with a ledger-based abstraction, providing a formal framework to serve as ground truth for attribution and clarifying guarantees for watermarking schemes.
We introduce ideal attribution mechanisms, a formal abstraction for reasoning about attribution decisions over strings. At the core of this abstraction lies the ledger, an append-only log of the prompt-response interaction history between a model and its user. Each mechanism produces deterministic decisions based on the ledger and an explicit selection criterion, making it well-suited to serve as a ground truth for attribution. We frame the design goal of watermarking schemes as faithful representation of ideal attribution mechanisms. This novel perspective brings conceptual clarity, replacing piecemeal probabilistic statements with a unified language for stating the guarantees of each scheme. It also enables precise reasoning about desiderata for future watermarking schemes, even when no current construction achieves them, since the ideal functionalities are specified first. In this way, the framework provides a roadmap that clarifies which guarantees are attainable in an idealized setting and worth pursuing in practice.