Diametrical Risk Minimization: Theory and Computations
This addresses a fundamental problem in machine learning optimization for researchers and practitioners dealing with nonconvex and sharp loss landscapes, though it appears incremental as an extension of ERM.
The paper tackles the issue of poor performance in Empirical Risk Minimization (ERM) due to large Lipschitz moduli and sharp minimizers by proposing Diametrical Risk Minimization (DRM), which achieves generalization bounds independent of Lipschitz moduli and finds low-error solutions in neural network classification with corrupted labels.
The theoretical and empirical performance of Empirical Risk Minimization (ERM) often suffers when loss functions are poorly behaved with large Lipschitz moduli and spurious sharp minimizers. We propose and analyze a counterpart to ERM called Diametrical Risk Minimization (DRM), which accounts for worst-case empirical risks within neighborhoods in parameter space. DRM has generalization bounds that are independent of Lipschitz moduli for convex as well as nonconvex problems and it can be implemented using a practical algorithm based on stochastic gradient descent. Numerical results illustrate the ability of DRM to find quality solutions with low generalization error in sharp empirical risk landscapes from benchmark neural network classification problems with corrupted labels.