Analysis of Generalized Bregman Surrogate Algorithms for Nonsmooth Nonconvex Statistical Learning
This work addresses optimization challenges in modern statistical applications, offering theoretical guarantees for a broad class of algorithms, though it appears incremental as it builds on existing Bregman-surrogate methods.
The paper tackles the problem of minimizing nonsmooth and nonconvex objective functions in high-dimensional statistical learning by analyzing a generalized Bregman-surrogate algorithm framework, establishing global convergence rates and showing that estimators approach statistical truth geometrically fast under certain conditions.
Modern statistical applications often involve minimizing an objective function that may be nonsmooth and/or nonconvex. This paper focuses on a broad Bregman-surrogate algorithm framework including the local linear approximation, mirror descent, iterative thresholding, DC programming and many others as particular instances. The recharacterization via generalized Bregman functions enables us to construct suitable error measures and establish global convergence rates for nonconvex and nonsmooth objectives in possibly high dimensions. For sparse learning problems with a composite objective, under some regularity conditions, the obtained estimators as the surrogate's fixed points, though not necessarily local minimizers, enjoy provable statistical guarantees, and the sequence of iterates can be shown to approach the statistical truth within the desired accuracy geometrically fast. The paper also studies how to design adaptive momentum based accelerations without assuming convexity or smoothness by carefully controlling stepsize and relaxation parameters.