A General Framework for Auditing Differentially Private Machine Learning
This work addresses the need for practical privacy verification in machine learning, which is crucial for ensuring compliance and security in real-world applications, though it builds incrementally on prior auditing methods.
The authors tackled the problem of auditing the privacy guarantees of differentially private machine learning implementations, developing a general framework that significantly improves auditing power over previous approaches on models like logistic regression, Naive Bayes, and random forest.
We present a framework to statistically audit the privacy guarantee conferred by a differentially private machine learner in practice. While previous works have taken steps toward evaluating privacy loss through poisoning attacks or membership inference, they have been tailored to specific models or have demonstrated low statistical power. Our work develops a general methodology to empirically evaluate the privacy of differentially private machine learning implementations, combining improved privacy search and verification methods with a toolkit of influence-based poisoning attacks. We demonstrate significantly improved auditing power over previous approaches on a variety of models including logistic regression, Naive Bayes, and random forest. Our method can be used to detect privacy violations due to implementation errors or misuse. When violations are not present, it can aid in understanding the amount of information that can be leaked from a given dataset, algorithm, and privacy specification.