TRSVR: An Adaptive Stochastic Trust-Region Method with Variance Reduction
This work addresses optimization efficiency for machine learning practitioners, but it is incremental as it builds on existing SVRG and trust-region techniques.
The authors tackled the problem of accelerating convergence in unconstrained nonconvex optimization by proposing a stochastic trust-region method with variance reduction, which achieves iteration and sample complexity bounds matching SVRG-based methods and outperforms SGD and Adam on machine learning tasks.
We propose a stochastic trust-region method for unconstrained nonconvex optimization that incorporates stochastic variance-reduced gradients (SVRG) to accelerate convergence. Unlike classical trust-region methods, the proposed algorithm relies solely on stochastic gradient information and does not require function value evaluations. The trust-region radius is adaptively adjusted based on a radius-control parameter and the stochastic gradient estimate. Under mild assumptions, we establish that the algorithm converges in expectation to a first-order stationary point. Moreover, the method achieves iteration and sample complexity bounds that match those of SVRG-based first-order methods, while allowing stochastic and potentially gradient-dependent second-order information. Extensive numerical experiments demonstrate that incorporating SVRG accelerates convergence, and that the use of trust-region methods and Hessian information further improves performance. We also highlight the impact of batch size and inner-loop length on efficiency, and show that the proposed method outperforms SGD and Adam on several machine learning tasks.