Generalization Bounds for Uniformly Stable Algorithms
This work provides stronger generalization guarantees for stable algorithms, benefiting theoretical machine learning researchers, but it is incremental as it refines existing bounds without new assumptions.
The paper tackles the problem of deriving meaningful generalization bounds for uniformly stable learning algorithms, particularly in settings where existing bounds are not tight, and achieves improved bounds of O(√((γ+1/n)log(1/δ))) and a tight second moment bound of O(γ²+1/n).
Uniform stability of a learning algorithm is a classical notion of algorithmic stability introduced to derive high-probability bounds on the generalization error (Bousquet and Elisseeff, 2002). Specifically, for a loss function with range bounded in $[0,1]$, the generalization error of a $γ$-uniformly stable learning algorithm on $n$ samples is known to be within $O((γ+1/n) \sqrt{n \log(1/δ)})$ of the empirical error with probability at least $1-δ$. Unfortunately, this bound does not lead to meaningful generalization bounds in many common settings where $γ\geq 1/\sqrt{n}$. At the same time the bound is known to be tight only when $γ= O(1/n)$. We substantially improve generalization bounds for uniformly stable algorithms without making any additional assumptions. First, we show that the bound in this setting is $O(\sqrt{(γ+ 1/n) \log(1/δ)})$ with probability at least $1-δ$. In addition, we prove a tight bound of $O(γ^2 + 1/n)$ on the second moment of the estimation error. The best previous bound on the second moment is $O(γ+ 1/n)$. Our proofs are based on new analysis techniques and our results imply substantially stronger generalization guarantees for several well-studied algorithms.