The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditing
It addresses the need for systematic evaluation of DP auditing methods for researchers and practitioners, but is incremental as it reviews and organizes existing work.
This paper systematizes research on auditing Differential Privacy (DP) techniques to identify key insights and open challenges, providing a reusable methodology to assess progress and future directions in the field.
This paper systematizes research on auditing Differential Privacy (DP) techniques, aiming to identify key insights into the current state of the art and open challenges. First, we introduce a comprehensive framework for reviewing work in the field and establish three cross-contextual desiderata that DP audits should target--namely, efficiency, end-to-end-ness, and tightness. Then, we systematize the modes of operation of state-of-the-art DP auditing techniques, including threat models, attacks, and evaluation functions. This allows us to highlight key details overlooked by prior work, analyze the limiting factors to achieving the three desiderata, and identify open research problems. Overall, our work provides a reusable and systematic methodology geared to assess progress in the field and identify friction points and future directions for our community to focus on.