Regularized $f$-Divergence Kernel Tests
This work addresses the need for robust statistical tests in machine learning applications such as differential privacy and unlearning, though it is incremental as it builds on existing kernel and divergence methods.
The authors tackled the problem of constructing practical kernel-based two-sample tests using f-divergences, resulting in a framework with theoretical guarantees for test power and adaptability to hyperparameters like kernel bandwidth and regularization. They demonstrated its utility in differential privacy auditing and machine unlearning evaluation, showing that different f-divergences detect localized differences and proposed a relative test for unlearning failures.
We propose a framework to construct practical kernel-based two-sample tests from the family of $f$-divergences. The test statistic is computed from the witness function of a regularized variational representation of the divergence, which we estimate using kernel methods. The proposed test is adaptive over hyperparameters such as the kernel bandwidth and the regularization parameter. We provide theoretical guarantees for statistical test power across our family of $f$-divergence estimates. While our test covers a variety of $f$-divergences, we bring particular focus to the Hockey-Stick divergence, motivated by its applications to differential privacy auditing and machine unlearning evaluation. For two-sample testing, experiments demonstrate that different $f$-divergences are sensitive to different localized differences, illustrating the importance of leveraging diverse statistics. For machine unlearning, we propose a relative test that distinguishes true unlearning failures from safe distributional variations.