A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
This work provides insights into the training dynamics of widely used optimization algorithms, with incremental improvements for scientific computing applications.
The study analyzed the dynamic behavior of RMSprop and Adam adaptive gradient algorithms, identifying three qualitative features in training loss curves: fast initial convergence, oscillations, and large spikes in the late phase, with Adam converging smoother and faster when momentum factors are close, particularly benefiting scientific computing tasks.
The dynamic behavior of RMSprop and Adam algorithms is studied through a combination of careful numerical experiments and theoretical explanations. Three types of qualitative features are observed in the training loss curve: fast initial convergence, oscillations, and large spikes in the late phase. The sign gradient descent (signGD) flow, which is the limit of Adam when taking the learning rate to 0 while keeping the momentum parameters fixed, is used to explain the fast initial convergence. For the late phase of Adam, three different types of qualitative patterns are observed depending on the choice of the hyper-parameters: oscillations, spikes, and divergence. In particular, Adam converges much smoother and even faster when the values of the two momentum factors are close to each other. This observation is particularly important for scientific computing tasks, for which the training process usually proceeds into the high precision regime.