SVGD: A Virtual Gradients Descent Method for Stochastic Optimization
This work addresses stochastic optimization for machine learning practitioners, but it appears incremental as it builds on existing gradient-based methods with a novel twist.
The authors tackled the problem of stochastic optimization by proposing the Stochastic Virtual Gradient Descent (SVGD) algorithm, which uses virtual gradients defined via computational graphs and automatic differentiation, resulting in computational efficiency and low memory usage, with experimental advantages over other methods on multiple datasets and network models.
Inspired by dynamic programming, we propose Stochastic Virtual Gradient Descent (SVGD) algorithm where the Virtual Gradient is defined by computational graph and automatic differentiation. The method is computationally efficient and has little memory requirements. We also analyze the theoretical convergence properties and implementation of the algorithm. Experimental results on multiple datasets and network models show that SVGD has advantages over other stochastic optimization methods.