Randomness and Interpolation Improve Gradient Descent
This work addresses optimization efficiency and overfitting in deep learning for researchers and practitioners, but it appears incremental as it builds directly on SGD without a major paradigm shift.
The paper tackled improving Stochastic Gradient Descent (SGD) by introducing two optimizers, IAGD and NRSGD, which use interpolation and noise regularization, respectively, and demonstrated their effectiveness on CIFAR-10 and CIFAR-100 datasets with CNNs.
Based on Stochastic Gradient Descent (SGD), the paper introduces two optimizers, named Interpolational Accelerating Gradient Descent (IAGD) as well as Noise-Regularized Stochastic Gradient Descent (NRSGD). IAGD leverages second-order Newton Interpolation to expedite the convergence process during training, assuming relevancy in gradients between iterations. To avoid over-fitting, NRSGD incorporates a noise regularization technique that introduces controlled noise to the gradients during the optimization process. Comparative experiments of this research are conducted on the CIFAR-10, and CIFAR-100 datasets, benchmarking different CNNs(Convolutional Neural Networks) with IAGD and NRSGD against classical optimizers in Keras Package. Results demonstrate the potential of those two viable improvement methods in SGD, implicating the effectiveness of the advancements.