Consistent Approximations in Composite Optimization
This provides a theoretical foundation for ensuring stability in optimization algorithms across various domains like machine learning and stochastic optimization, but it is incremental as it builds on existing approximation concepts.
The paper tackles the problem of approximation errors in composite optimization, which can lead to large solution errors, by developing a framework of consistent approximations that ensures well-behaved minimizers, stationary points, and level-sets for a broad class of non-convex and non-smooth problems, with quantitative analysis providing convergence rates.
Approximations of optimization problems arise in computational procedures and sensitivity analysis. The resulting effect on solutions can be significant, with even small approximations of components of a problem translating into large errors in the solutions. We specify conditions under which approximations are well behaved in the sense of minimizers, stationary points, and level-sets and this leads to a framework of consistent approximations. The framework is developed for a broad class of composite problems, which are neither convex nor smooth. We demonstrate the framework using examples from stochastic optimization, neural-network based machine learning, distributionally robust optimization, penalty and augmented Lagrangian methods, interior-point methods, homotopy methods, smoothing methods, extended nonlinear programming, difference-of-convex programming, and multi-objective optimization. An enhanced proximal method illustrates the algorithmic possibilities. A quantitative analysis supplements the development by furnishing rates of convergence.