Analyzing Information Leakage of Updates to Natural Language Models
This work addresses privacy risks for users of machine learning systems that require regular model updates, though it is incremental as it builds on existing differential privacy and model analysis concepts.
The authors tackled the problem of information leakage from updates to natural language models by showing that differential analysis of model snapshots can reveal detailed information about changes in training data, proposing new metrics for analysis and evaluating mitigation strategies.
To continuously improve quality and reflect changes in data, machine learning applications have to regularly retrain and update their core models. We show that a differential analysis of language model snapshots before and after an update can reveal a surprising amount of detailed information about changes in the training data. We propose two new metrics---\emph{differential score} and \emph{differential rank}---for analyzing the leakage due to updates of natural language models. We perform leakage analysis using these metrics across models trained on several different datasets using different methods and configurations. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.