Bennett-type Generalization Bounds: Large-deviation Case and Faster Rate of Convergence
This work provides improved theoretical guarantees for generalization in machine learning, though it appears incremental as it builds on existing Bennett-type bounds.
The paper tackles the problem of deriving generalization bounds for learning processes with i.i.d. samples, achieving a faster convergence rate of o(N^{-1/2}) compared to traditional O(N^{-1/2}) results, particularly in large-deviation cases where empirical and expected risks differ significantly.
In this paper, we present the Bennett-type generalization bounds of the learning process for i.i.d. samples, and then show that the generalization bounds have a faster rate of convergence than the traditional results. In particular, we first develop two types of Bennett-type deviation inequality for the i.i.d. learning process: one provides the generalization bounds based on the uniform entropy number; the other leads to the bounds based on the Rademacher complexity. We then adopt a new method to obtain the alternative expressions of the Bennett-type generalization bounds, which imply that the bounds have a faster rate o(N^{-1/2}) of convergence than the traditional results O(N^{-1/2}). Additionally, we find that the rate of the bounds will become faster in the large-deviation case, which refers to a situation where the empirical risk is far away from (at least not close to) the expected risk. Finally, we analyze the asymptotical convergence of the learning process and compare our analysis with the existing results.