Zeroth-Order Methods for Stochastic Nonconvex Nonsmooth Composite Optimization
This work addresses optimization problems for machine learning applications like regularized ReLU networks and sparse support matrix machines, where previous methods fail due to smoothness requirements, though it is incremental in extending zeroth-order methods to a broader class of problems.
The paper tackles stochastic nonconvex nonsmooth composite optimization without smoothness assumptions, proposing two new notions of approximate stationary points and achieving finite-time convergence results for two zeroth-order algorithms, with effectiveness demonstrated through numerical experiments.
This work aims to solve a stochastic nonconvex nonsmooth composite optimization problem. Previous works on composite optimization problem requires the major part to satisfy Lipschitz smoothness or some relaxed smoothness conditions, which excludes some machine learning examples such as regularized ReLU network and sparse support matrix machine. In this work, we focus on stochastic nonconvex composite optimization problem without any smoothness assumptions. In particular, we propose two new notions of approximate stationary points for such optimization problem and obtain finite-time convergence results of two zeroth-order algorithms to these two approximate stationary points respectively. Finally, we demonstrate that these algorithms are effective using numerical experiments.