LaProp: Separating Momentum and Adaptivity in Adam
This addresses instability issues in optimization for machine learning practitioners, though it is incremental as it builds on existing Adam-style methods.
The authors tackled the problem of unnecessary coupling between momentum and adaptivity in Adam-style optimizers, which causes instability and divergence, by proposing LaProp to decouple them, resulting in consistently improved speed and stability over Adam on various tasks.
We identity a by-far-unrecognized problem of Adam-style optimizers which results from unnecessary coupling between momentum and adaptivity. The coupling leads to instability and divergence when the momentum and adaptivity parameters are mismatched. In this work, we propose a method, Laprop, which decouples momentum and adaptivity in the Adam-style methods. We show that the decoupling leads to greater flexibility in the hyperparameters and allows for a straightforward interpolation between the signed gradient methods and the adaptive gradient methods. We experimentally show that Laprop has consistently improved speed and stability over Adam on a variety of tasks. We also bound the regret of Laprop on a convex problem and show that our bound differs from that of Adam by a key factor, which demonstrates its advantage.