Memorization for Good: Encryption with Autoregressive Language Models
This work addresses the challenge of secure data encryption for AI and cryptography communities by repurposing a typically undesirable property of language models, though it appears incremental as it builds on existing memorization capabilities.
The paper tackles the problem of leveraging memorization in language models for encryption, proposing SELM, a symmetric encryption algorithm that encodes data into a compact vector and losslessly decodes it, with promising security results in empirical tests.
Over-parameterized neural language models (LMs) can memorize and recite long sequences of training data. While such memorization is normally associated with undesired properties such as overfitting and information leaking, our work casts memorization as an unexplored capability of LMs. We propose the first symmetric encryption algorithm with autoregressive language models (SELM). We show that autoregressive LMs can encode arbitrary data into a compact real-valued vector (i.e., encryption) and then losslessly decode the vector to the original message (i.e., decryption) via random subspace optimization and greedy decoding. While SELM is not amenable to conventional cryptanalysis, we investigate its security through a novel empirical variant of the classic IND-CPA (indistinguishability under chosen-plaintext attack) game and show promising results on security. Our code and datasets are available at https://github.com/OSU-NLP-Group/SELM.