Measuring Unintended Memorisation of Unique Private Features in Neural Networks
This highlights a privacy risk for sensitive data in applications like healthcare, where patient information could be leaked, though it is incremental as it builds on known memorization issues.
The paper tackled the problem of neural networks unintentionally memorizing unique private features from training data, such as a person's name in an image, and found that this occurs even with strategies like early stopping or regularization, posing a privacy risk for rare information.
Neural networks pose a privacy risk to training data due to their propensity to memorise and leak information. Focusing on image classification, we show that neural networks also unintentionally memorise unique features even when they occur only once in training data. An example of a unique feature is a person's name that is accidentally present on a training image. Assuming access to the inputs and outputs of a trained model, the domain of the training data, and knowledge of unique features, we develop a score estimating the model's sensitivity to a unique feature by comparing the KL divergences of the model's output distributions given modified out-of-distribution images. Our results suggest that unique features are memorised by multi-layer perceptrons and convolutional neural networks trained on benchmark datasets, such as MNIST, Fashion-MNIST and CIFAR-10. We find that strategies to prevent overfitting (e.g.\ early stopping, regularisation, batch normalisation) do not prevent memorisation of unique features. These results imply that neural networks pose a privacy risk to rarely occurring private information. These risks can be more pronounced in healthcare applications if patient information is present in the training data.