The Test of Tests: A Framework For Differentially Private Hypothesis Testing
This work provides a practical solution for statisticians and data analysts needing privacy-preserving hypothesis testing, though it is incremental as it builds on existing frameworks.
The authors tackled the problem of creating differentially private versions of any hypothesis test in a black-box way, showing that at epsilon = 1, their framework only requires 5-6 times as much data as in the fully public setting while achieving higher power than other generic solutions and often outperforming individually-designed tests.
We present a generic framework for creating differentially private versions of any hypothesis test in a black-box way. We analyze the resulting tests analytically and experimentally. Most crucially, we show good practical performance for small data sets, showing that at epsilon = 1 we only need 5-6 times as much data as in the fully public setting. We compare our work to the one existing framework of this type, as well as to several individually-designed private hypothesis tests. Our framework is higher power than other generic solutions and at least competitive with (and often better than) individually-designed tests.