Auditing $f$-Differential Privacy in One Run
This work addresses the need for more efficient and accurate privacy auditing in machine learning, which is crucial for ensuring the reliability of privacy-preserving implementations, though it appears incremental by building on prior methods.
The paper tackles the problem of inefficient and suboptimal empirical auditing of privacy-preserving algorithms by introducing a tight and efficient auditing procedure that requires only a single run of the target mechanism, leveraging dataset randomness and using hypothesized f-DP curves for more accurate privacy estimates.
Empirical auditing has emerged as a means of catching some of the flaws in the implementation of privacy-preserving algorithms. Existing auditing mechanisms, however, are either computationally inefficient requiring multiple runs of the machine learning algorithms or suboptimal in calculating an empirical privacy. In this work, we present a tight and efficient auditing procedure and analysis that can effectively assess the privacy of mechanisms. Our approach is efficient; similar to the recent work of Steinke, Nasr, and Jagielski (2023), our auditing procedure leverages the randomness of examples in the input dataset and requires only a single run of the target mechanism. And it is more accurate; we provide a novel analysis that enables us to achieve tight empirical privacy estimates by using the hypothesized $f$-DP curve of the mechanism, which provides a more accurate measure of privacy than the traditional $ε,δ$ differential privacy parameters. We use our auditing procure and analysis to obtain empirical privacy, demonstrating that our auditing procedure delivers tighter privacy estimates.