GPT, But Backwards: Exactly Inverting Language Model Outputs
This work addresses security vulnerabilities in language models, such as stealing prompts and leaking data, by enabling exact inversion, though it is incremental as it builds on existing inversion methods.
The authors tackled the problem of reconstructing unknown textual inputs from language model outputs, presenting the SODA algorithm which achieved 98% and 79% reconstruction rates on inputs up to 10 tokens for natural language and random inputs, respectively.
The task of reconstructing unknown textual inputs to language models is a fundamental auditing primitive that allows us to assess the model's vulnerability to a range of security issues, including stealing hidden system prompts, detecting backdoors, and leaking private data. Existing inversion works assume access to differing levels of information (e.g. requiring input-output examples, the model parameters, intermediate activations or output logits) but oftentimes fail to fully reconstruct the desired input. In this paper, we present the Sparse One-hot Discrete Adam (SODA) algorithm, a search-based inversion method that can accurately reconstruct the input text, given white-box access to the language model and its output. Our experiments demonstrate for the first time that exact language model inversion is possible on both natural language and random inputs. Indeed, SODA achieves respectively 98% and 79% reconstruction rates on inputs with lengths up to 10 tokens. Furthermore, we show that input length and vocabulary size have a far greater impact on the probability of a successful reconstruction than the size of the language model itself, thus allowing us to scale to models from 33M to 3B parameters.