Leave-one-out Distinguishability in Machine Learning
This work addresses privacy and security concerns in machine learning by providing a framework to assess data memorization and leakage risks, which is incremental as it refines existing measures.
The authors tackled the problem of quantifying how individual training data points affect a machine learning model's output, introducing leave-one-out distinguishability (LOOD) to measure memorization, information leakage, and data influence, with validation through empirical analysis of membership inference attacks and demonstration of its use for accurate training data reconstruction.
We introduce an analytical framework to quantify the changes in a machine learning algorithm's output distribution following the inclusion of a few data points in its training set, a notion we define as leave-one-out distinguishability (LOOD). This is key to measuring data **memorization** and information **leakage** as well as the **influence** of training data points in machine learning. We illustrate how our method broadens and refines existing empirical measures of memorization and privacy risks associated with training data. We use Gaussian processes to model the randomness of machine learning algorithms, and validate LOOD with extensive empirical analysis of leakage using membership inference attacks. Our analytical framework enables us to investigate the causes of leakage and where the leakage is high. For example, we analyze the influence of activation functions, on data memorization. Additionally, our method allows us to identify queries that disclose the most information about the training data in the leave-one-out setting. We illustrate how optimal queries can be used for accurate **reconstruction** of training data.