Enhancing One-run Privacy Auditing with Quantile Regression-Based Membership Inference
This work addresses the challenge of practical privacy auditing for machine learning practitioners, but it is incremental as it builds on existing one-run methods by incorporating stronger attacks.
The paper tackled the problem of improving one-run privacy auditing for differential privacy mechanisms like DP-SGD in black-box settings, where prior methods had large gaps between empirical and theoretical bounds. The result showed that using quantile regression for membership inference attacks achieved tighter bounds while maintaining computational efficiency, as evaluated on CIFAR-10 image classification models.
Differential privacy (DP) auditing aims to provide empirical lower bounds on the privacy guarantees of DP mechanisms like DP-SGD. While some existing techniques require many training runs that are prohibitively costly, recent work introduces one-run auditing approaches that effectively audit DP-SGD in white-box settings while still being computationally efficient. However, in the more practical black-box setting where gradients cannot be manipulated during training and only the last model iterate is observed, prior work shows that there is still a large gap between the empirical lower bounds and theoretical upper bounds. Consequently, in this work, we study how incorporating approaches for stronger membership inference attacks (MIA) can improve one-run auditing in the black-box setting. Evaluating on image classification models trained on CIFAR-10 with DP-SGD, we demonstrate that our proposed approach, which utilizes quantile regression for MIA, achieves tighter bounds while crucially maintaining the computational efficiency of one-run methods.