Variance-reduced first-order methods for deterministically constrained stochastic nonconvex optimization with strong convergence guarantees
This addresses the need for reliable constraint satisfaction in practical applications like engineering or finance, offering an incremental improvement over existing methods by tightening guarantees.
The paper tackles the problem of deterministically constrained stochastic nonconvex optimization by proposing variance-reduced first-order methods that ensure constraints are nearly satisfied with certainty, achieving sample and operation complexities of up to O(ε^{-3}) and O(ε^{-4}) for a stronger stationary point.
In this paper, we study a class of deterministically constrained stochastic optimization problems. Existing methods typically aim to find an $ε$-stochastic stationary point, where the expected violations of both constraints and first-order stationarity are within a prescribed accuracy $ε$. However, in many practical applications, it is crucial that the constraints be nearly satisfied with certainty, making such an $ε$-stochastic stationary point potentially undesirable due to the risk of significant constraint violations. To address this issue, we propose single-loop variance-reduced stochastic first-order methods, where the stochastic gradient of the stochastic component is computed using either a truncated recursive momentum scheme or a truncated Polyak momentum scheme for variance reduction, while the gradient of the deterministic component is computed exactly. Under the error bound condition with a parameter $θ\geq 1$ and other suitable assumptions, we establish that these methods respectively achieve a sample and first-order operation complexity of $\widetilde O(ε^{-\max\{θ+2, 2θ\}})$ and $\widetilde O(ε^{-\max\{4, 2θ\}})$ for finding a stronger $ε$-stochastic stationary point, where the constraint violation is within $ε$ with certainty, and the expected violation of first-order stationarity is within $ε$. For $θ=1$, these complexities reduce to $\widetilde O(ε^{-3})$ and $\widetilde O(ε^{-4})$ respectively, which match, up to a logarithmic factor, the best-known complexities achieved by existing methods for finding an $ε$-stochastic stationary point of unconstrained smooth stochastic optimization problems.