PAC learning with stable and private predictions
This work addresses the challenge of reducing sample overhead for stable and private learning, which is crucial for applications requiring data privacy and robustness, though it is incremental as it builds on existing notions.
The paper tackles the problem of achieving stable and private predictions in binary classification with reduced sample complexity, demonstrating algorithms that improve bounds from previous work, such as a γ-uniformly stable algorithm using Õ(d/(αγ) + d/α²) samples and nearly matching lower bounds.
We study binary classification algorithms for which the prediction on any point is not too sensitive to individual examples in the dataset. Specifically, we consider the notions of uniform stability (Bousquet and Elisseeff, 2001) and prediction privacy (Dwork and Feldman, 2018). Previous work on these notions shows how they can be achieved in the standard PAC model via simple aggregation of models trained on disjoint subsets of data. Unfortunately, this approach leads to a significant overhead in terms of sample complexity. Here we demonstrate several general approaches to stable and private prediction that either eliminate or significantly reduce the overhead. Specifically, we demonstrate that for any class $C$ of VC dimension $d$ there exists a $γ$-uniformly stable algorithm for learning $C$ with excess error $α$ using $\tilde O(d/(αγ) + d/α^2)$ samples. We also show that this bound is nearly tight. For $ε$-differentially private prediction we give two new algorithms: one using $\tilde O(d/(α^2ε))$ samples and another one using $\tilde O(d^2/(αε) + d/α^2)$ samples. The best previously known bounds for these problems are $O(d/(α^2γ))$ and $O(d/(α^3ε))$, respectively.