Efficient ANN-Guided Distillation: Aligning Rate-based Features of Spiking Neural Networks through Hybrid Block-wise Replacement
This work addresses the problem of efficiently training SNNs for researchers and practitioners in neuromorphic computing, though it appears incremental as it builds on existing ANN-to-SNN conversion and distillation techniques.
The paper tackles the challenge of training Spiking Neural Networks (SNNs) by proposing an ANN-SNN distillation framework that uses a block-wise replacement strategy to align SNN features with ANN features, achieving results comparable to or better than state-of-the-art SNN distillation methods with improved efficiency.
Spiking Neural Networks (SNNs) have garnered considerable attention as a potential alternative to Artificial Neural Networks (ANNs). Recent studies have highlighted SNNs' potential on large-scale datasets. For SNN training, two main approaches exist: direct training and ANN-to-SNN (ANN2SNN) conversion. To fully leverage existing ANN models in guiding SNN learning, either direct ANN-to-SNN conversion or ANN-SNN distillation training can be employed. In this paper, we propose an ANN-SNN distillation framework from the ANN-to-SNN perspective, designed with a block-wise replacement strategy for ANN-guided learning. By generating intermediate hybrid models that progressively align SNN feature spaces to those of ANN through rate-based features, our framework naturally incorporates rate-based backpropagation as a training method. Our approach achieves results comparable to or better than state-of-the-art SNN distillation methods, showing both training and learning efficiency.