The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks
This addresses privacy risks for users of machine learning models trained on sensitive data, offering a practical defense against data leakage.
The paper tackles the problem of unintended memorization of sensitive data by generative sequence models, developing a testing methodology to assess and mitigate this risk, and demonstrates its application by extracting secret sequences like credit card numbers and limiting data exposure in Google's Smart Compose.
This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.