Online Curvature-Aware Replay: Leveraging $\mathbf{2^{nd}}$ Order Information for Online Continual Learning
This addresses the challenge of maintaining stability and preventing forgetting in online continual learning for AI systems that adapt to nonstationary data streams, representing a strong incremental improvement over existing replay-based methods.
The paper tackled the problem of instability in online continual learning models after task shifts by formalizing replay-based methods as a second-order optimization with KL-divergence constraints, and proposed OCAR, which uses a K-FAC approximation of the Fisher Information Matrix to precondition gradients, resulting in higher average accuracy across three benchmarks.
Online Continual Learning (OCL) models continuously adapt to nonstationary data streams, usually without task information. These settings are complex and many traditional CL methods fail, while online methods (mainly replay-based) suffer from instabilities after the task shift. To address this issue, we formalize replay-based OCL as a second-order online joint optimization with explicit KL-divergence constraints on replay data. We propose Online Curvature-Aware Replay (OCAR) to solve the problem: a method that leverages second-order information of the loss using a K-FAC approximation of the Fisher Information Matrix (FIM) to precondition the gradient. The FIM acts as a stabilizer to prevent forgetting while also accelerating the optimization in non-interfering directions. We show how to adapt the estimation of the FIM to a continual setting stabilizing second-order optimization for non-iid data, uncovering the role of the Tikhonov regularization in the stability-plasticity tradeoff. Empirical results show that OCAR outperforms state-of-the-art methods in continual metrics achieving higher average accuracy throughout the training process in three different benchmarks.