Adaptive Optimization with Examplewise Gradients
This is an incremental improvement for machine learning practitioners, addressing optimization efficiency in typical setups.
The authors tackled the problem of stochastic gradient optimization by proposing a framework where optimizers access batches of gradient estimates per iteration instead of single estimates, and developed Eve as an adaptation of Adam using examplewise gradients. Preliminary experiments without hyperparameter tuning showed Eve slightly outperformed Adam on a small-scale benchmark but performed the same or worse on larger-scale benchmarks.
We propose a new, more general approach to the design of stochastic gradient-based optimization methods for machine learning. In this new framework, optimizers assume access to a batch of gradient estimates per iteration, rather than a single estimate. This better reflects the information that is actually available in typical machine learning setups. To demonstrate the usefulness of this generalized approach, we develop Eve, an adaptation of the Adam optimizer which uses examplewise gradients to obtain more accurate second-moment estimates. We provide preliminary experiments, without hyperparameter tuning, which show that the new optimizer slightly outperforms Adam on a small scale benchmark and performs the same or worse on larger scale benchmarks. Further work is needed to refine the algorithm and tune hyperparameters.