Rethinking Benchmarks for Differentially Private Image Classification
This work addresses the need for standardized evaluation in differentially private machine learning for researchers, though it is incremental as it builds on existing techniques rather than introducing new methods.
The authors tackled the problem of inconsistent evaluation in differentially private image classification by proposing comprehensive benchmarks and a public leaderboard, testing established techniques across various settings to identify which remain effective.
We revisit benchmarks for differentially private image classification. We suggest a comprehensive set of benchmarks, allowing researchers to evaluate techniques for differentially private machine learning in a variety of settings, including with and without additional data, in convex settings, and on a variety of qualitatively different datasets. We further test established techniques on these benchmarks in order to see which ideas remain effective in different settings. Finally, we create a publicly available leader board for the community to track progress in differentially private machine learning.