Towards Optimal Problem Dependent Generalization Error Bounds in Statistical Learning Theory
This work addresses fundamental limitations in statistical learning theory by providing improved and unified bounds for generalization error, which is crucial for researchers and practitioners in machine learning seeking optimal performance guarantees.
The paper tackles the problem of deriving optimal problem-dependent generalization error bounds in statistical learning theory, introducing a 'uniform localized convergence' framework that achieves sharp rates for various learning problems, including providing the first estimator to achieve optimal variance-dependent rates for rich classes and showing iterative algorithms can achieve optimal generalization error in non-convex learning and other areas.
We study problem-dependent rates, i.e., generalization errors that scale near-optimally with the variance, the effective loss, or the gradient norms evaluated at the "best hypothesis." We introduce a principled framework dubbed "uniform localized convergence," and characterize sharp problem-dependent rates for central statistical learning problems. From a methodological viewpoint, our framework resolves several fundamental limitations of existing uniform convergence and localization analysis approaches. It also provides improvements and some level of unification in the study of localized complexities, one-sided uniform inequalities, and sample-based iterative algorithms. In the so-called "slow rate" regime, we provides the first (moment-penalized) estimator that achieves the optimal variance-dependent rate for general "rich" classes; we also establish improved loss-dependent rate for standard empirical risk minimization. In the "fast rate" regime, we establish finite-sample problem-dependent bounds that are comparable to precise asymptotics. In addition, we show that iterative algorithms like gradient descent and first-order Expectation-Maximization can achieve optimal generalization error in several representative problems across the areas of non-convex learning, stochastic optimization, and learning with missing data.