Graphical Convergence of Subgradients in Nonconvex Optimization and Learning
This work addresses convergence issues in nonconvex optimization for machine learning, providing theoretical guarantees for subgradient estimation in nonsmooth settings.
The paper tackles the problem of estimating subgradients for weakly convex losses in stochastic optimization, establishing dimension-dependent rates for general cases and dimension-independent rates for generalized linear models.
We investigate the stochastic optimization problem of minimizing population risk, where the loss defining the risk is assumed to be weakly convex. Compositions of Lipschitz convex functions with smooth maps are the primary examples of such losses. We analyze the estimation quality of such nonsmooth and nonconvex problems by their sample average approximations. Our main results establish dimension-dependent rates on subgradient estimation in full generality and dimension-independent rates when the loss is a generalized linear model. As an application of the developed techniques, we analyze the nonsmooth landscape of a robust nonlinear regression problem.