Elephant Neural Networks: Born to Be a Continual Learner
This addresses catastrophic forgetting in neural networks for continual learning applications, offering a novel architectural solution with broad applicability.
The study tackled catastrophic forgetting in continual learning by investigating the role of activation functions, revealing that gradient sparsity reduces forgetting, and proposed elephant activation functions that improve resilience without replay buffers or task boundaries, achieving excellent performance on Split MNIST in a single pass.
Catastrophic forgetting remains a significant challenge to continual learning for decades. While recent works have proposed effective methods to mitigate this problem, they mainly focus on the algorithmic side. Meanwhile, we do not fully understand what architectural properties of neural networks lead to catastrophic forgetting. This study aims to fill this gap by studying the role of activation functions in the training dynamics of neural networks and their impact on catastrophic forgetting. Our study reveals that, besides sparse representations, the gradient sparsity of activation functions also plays an important role in reducing forgetting. Based on this insight, we propose a new class of activation functions, elephant activation functions, that can generate both sparse representations and sparse gradients. We show that by simply replacing classical activation functions with elephant activation functions, we can significantly improve the resilience of neural networks to catastrophic forgetting. Our method has broad applicability and benefits for continual learning in regression, class incremental learning, and reinforcement learning tasks. Specifically, we achieves excellent performance on Split MNIST dataset in just one single pass, without using replay buffer, task boundary information, or pre-training.