Enhancing Efficient Continual Learning with Dynamic Structure Development of Spiking Neural Networks
This addresses the problem of energy-efficient continual learning for artificial general intelligence, though it is incremental as it builds on existing SNN frameworks.
The paper tackles efficient continual learning by proposing a dynamic structure development method for spiking neural networks (DSD-SNN), which dynamically grows and prunes neurons to handle sequential tasks, resulting in improved performance, learning speed, memory capacity, and reduced computational overhead, with comparable performance to DNN-based methods and outperforming existing SNN-based SOTA.
Children possess the ability to learn multiple cognitive tasks sequentially, which is a major challenge toward the long-term goal of artificial general intelligence. Existing continual learning frameworks are usually applicable to Deep Neural Networks (DNNs) and lack the exploration on more brain-inspired, energy-efficient Spiking Neural Networks (SNNs). Drawing on continual learning mechanisms during child growth and development, we propose Dynamic Structure Development of Spiking Neural Networks (DSD-SNN) for efficient and adaptive continual learning. When learning a sequence of tasks, the DSD-SNN dynamically assigns and grows new neurons to new tasks and prunes redundant neurons, thereby increasing memory capacity and reducing computational overhead. In addition, the overlapping shared structure helps to quickly leverage all acquired knowledge to new tasks, empowering a single network capable of supporting multiple incremental tasks (without the separate sub-network mask for each task). We validate the effectiveness of the proposed model on multiple class incremental learning and task incremental learning benchmarks. Extensive experiments demonstrated that our model could significantly improve performance, learning speed and memory capacity, and reduce computational overhead. Besides, our DSD-SNN model achieves comparable performance with the DNNs-based methods, and significantly outperforms the state-of-the-art (SOTA) performance for existing SNNs-based continual learning methods.