A Stochastic Gradient Method with Biased Estimation for Faster Nonconvex Optimization
This work addresses optimization efficiency for nonconvex problems in machine learning, offering incremental improvements in convergence speed.
The paper tackles the problem of optimizing nonconvex objectives, such as deep learning models, by proposing an integrated approach that balances biased and unbiased gradient estimation to improve convergence rates, with theoretical and experimental validation showing configurable hyper-parameters enhance performance.
A number of optimization approaches have been proposed for optimizing nonconvex objectives (e.g. deep learning models), such as batch gradient descent, stochastic gradient descent and stochastic variance reduced gradient descent. Theory shows these optimization methods can converge by using an unbiased gradient estimator. However, in practice biased gradient estimation can allow more efficient convergence to the vicinity since an unbiased approach is computationally more expensive. To produce fast convergence there are two trade-offs of these optimization strategies which are between stochastic/batch, and between biased/unbiased. This paper proposes an integrated approach which can control the nature of the stochastic element in the optimizer and can balance the trade-off of estimator between the biased and unbiased by using a hyper-parameter. It is shown theoretically and experimentally that this hyper-parameter can be configured to provide an effective balance to improve the convergence rate.