Developmental Plasticity-inspired Adaptive Pruning for Deep Spiking and Artificial Neural Networks
This work addresses the need for more efficient and biologically plausible compression methods in deep learning, particularly for spiking neural networks, though it appears incremental by building on existing pruning concepts with biological inspiration.
The paper tackled the problem of inefficient network compression in deep artificial and spiking neural networks by proposing a developmental plasticity-inspired adaptive pruning method, which dynamically optimizes network structures during learning without pre-training or retraining, achieving state-of-the-art performance with extremely compressed networks on diverse benchmarks.
Developmental plasticity plays a prominent role in shaping the brain's structure during ongoing learning in response to dynamically changing environments. However, the existing network compression methods for deep artificial neural networks (ANNs) and spiking neural networks (SNNs) draw little inspiration from brain's developmental plasticity mechanisms, thus limiting their ability to learn efficiently, rapidly, and accurately. This paper proposed a developmental plasticity-inspired adaptive pruning (DPAP) method, with inspiration from the adaptive developmental pruning of dendritic spines, synapses, and neurons according to the ``use it or lose it, gradually decay" principle. The proposed DPAP model considers multiple biologically realistic mechanisms (such as dendritic spine dynamic plasticity, activity-dependent neural spiking trace, and local synaptic plasticity), with additional adaptive pruning strategy, so that the network structure can be dynamically optimized during learning without any pre-training and retraining. Extensive comparative experiments show consistent and remarkable performance and speed boost with the extremely compressed networks on a diverse set of benchmark tasks for deep ANNs and SNNs, especially the spatio-temporal joint pruning of SNNs in neuromorphic datasets. This work explores how developmental plasticity enables complex deep networks to gradually evolve into brain-like efficient and compact structures, eventually achieving state-of-the-art (SOTA) performance for biologically realistic SNNs.