Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models
This work addresses a theoretical problem in machine learning by providing exact comparisons for interpolators, offering insights into generalization beyond linear models, though it is incremental as it builds on prior work.
The paper tackles the gap between uniform convergence bounds and test error in interpolating predictors, deriving exact analytical expressions for these bounds in random feature models and showing that uniform convergence over interpolators can provide non-trivial or improved bounds even when classical bounds are vacuous.
Recent work showed that there could be a large gap between the classical uniform convergence bound and the actual test error of zero-training-error predictors (interpolators) such as deep neural networks. To better understand this gap, we study the uniform convergence in the nonlinear random feature model and perform a precise theoretical analysis on how uniform convergence depends on the sample size and the number of parameters. We derive and prove analytical expressions for three quantities in this model: 1) classical uniform convergence over norm balls, 2) uniform convergence over interpolators in the norm ball (recently proposed by Zhou et al. (2020)), and 3) the risk of minimum norm interpolator. We show that, in the setting where the classical uniform convergence bound is vacuous (diverges to $\infty$), uniform convergence over the interpolators still gives a non-trivial bound of the test error of interpolating solutions. We also showcase a different setting where classical uniform convergence bound is non-vacuous, but uniform convergence over interpolators can give an improved sample complexity guarantee. Our result provides a first exact comparison between the test errors and uniform convergence bounds for interpolators beyond simple linear models.