Generalization Bounds for Label Noise Stochastic Gradient Descent
This provides theoretical insights into the effect of label noise in optimization, though it is incremental as it builds on existing stability frameworks.
The paper tackles the problem of analyzing generalization error for stochastic gradient descent with label noise in non-convex settings, achieving a bound that scales polynomially with dimension and at a rate of n^{-2/3}, which improves upon the prior n^{-1/2} rate for SGLD.
We develop generalization error bounds for stochastic gradient descent (SGD) with label noise in non-convex settings under uniform dissipativity and smoothness conditions. Under a suitable choice of semimetric, we establish a contraction in Wasserstein distance of the label noise stochastic gradient flow that depends polynomially on the parameter dimension $d$. Using the framework of algorithmic stability, we derive time-independent generalisation error bounds for the discretized algorithm with a constant learning rate. The error bound we achieve scales polynomially with $d$ and with the rate of $n^{-2/3}$, where $n$ is the sample size. This rate is better than the best-known rate of $n^{-1/2}$ established for stochastic gradient Langevin dynamics (SGLD) -- which employs parameter-independent Gaussian noise -- under similar conditions. Our analysis offers quantitative insights into the effect of label noise.