Fisher-Orthogonal Projected Natural Gradient Descent for Continual Learning
This addresses the problem of enabling neural networks to learn sequentially without forgetting for AI systems that require lifelong learning, though it is incremental as it builds on existing orthogonal gradient and natural gradient methods.
The paper tackles catastrophic forgetting in continual learning by proposing the FOPNG optimizer, which enforces Fisher-orthogonal constraints on parameter updates to preserve old task performance while learning new tasks, demonstrating strong results on standard benchmarks like Permuted-MNIST and Split-CIFAR100.
Continual learning aims to enable neural networks to acquire new knowledge on sequential tasks. However, the key challenge in such settings is to learn new tasks without catastrophically forgetting previously learned tasks. We propose the Fisher-Orthogonal Projected Natural Gradient Descent (FOPNG) optimizer, which enforces Fisher-orthogonal constraints on parameter updates to preserve old task performance while learning new tasks. Unlike existing methods that operate in Euclidean parameter space, FOPNG projects gradients onto the Fisher-orthogonal complement of previous task gradients. This approach unifies natural gradient descent with orthogonal gradient methods within an information-geometric framework. The resulting update direction is invariant under reparameterization, guarantees descent in the Fisher metric, and helps preserve prior task outputs. We provide theoretical analysis establishing the properties of the projected update, describe efficient and practical implementations using the diagonal Fisher, and demonstrate strong results on standard continual learning benchmarks such as Permuted-MNIST, Split-MNIST, Rotated-MNIST, Split-CIFAR10, and Split-CIFAR100.