DualNet: Continual Learning, Fast and Slow
This work addresses continual learning for AI systems, offering a novel approach that improves robustness and scalability, though it is incremental in building on existing neuroscience-inspired methods.
The authors tackled the problem of continual learning by proposing DualNet, a framework inspired by neuroscience that combines fast supervised learning for task-specific representations with slow unsupervised learning for general representations, achieving state-of-the-art performance on benchmarks like CORE50 and miniImageNet with significant margins.
According to Complementary Learning Systems (CLS) theory~\citep{mcclelland1995there} in neuroscience, humans do effective \emph{continual learning} through two complementary systems: a fast learning system centered on the hippocampus for rapid learning of the specifics and individual experiences, and a slow learning system located in the neocortex for the gradual acquisition of structured knowledge about the environment. Motivated by this theory, we propose a novel continual learning framework named "DualNet", which comprises a fast learning system for supervised learning of pattern-separated representation from specific tasks and a slow learning system for unsupervised representation learning of task-agnostic general representation via a Self-Supervised Learning (SSL) technique. The two fast and slow learning systems are complementary and work seamlessly in a holistic continual learning framework. Our extensive experiments on two challenging continual learning benchmarks of CORE50 and miniImageNet show that DualNet outperforms state-of-the-art continual learning methods by a large margin. We further conduct ablation studies of different SSL objectives to validate DualNet's efficacy, robustness, and scalability. Code will be made available upon acceptance.