Time-step Mixup for Efficient Spiking Knowledge Transfer from Appearance to Event Domain
This work addresses the problem of limited event data and modality gaps for researchers in energy-efficient visual processing, offering an incremental improvement over prior transfer methods.
The paper tackles the challenge of transferring knowledge from RGB to event camera data for spiking neural networks by proposing Time-step Mixup knowledge transfer (TMKT), which interpolates inputs across time-steps and uses modality-aware objectives, achieving superior performance in spiking image classification tasks.
The integration of event cameras and spiking neural networks holds great promise for energy-efficient visual processing. However, the limited availability of event data and the sparse nature of DVS outputs pose challenges for effective training. Although some prior work has attempted to transfer semantic knowledge from RGB datasets to DVS, they often overlook the significant distribution gap between the two modalities. In this paper, we propose Time-step Mixup knowledge transfer (TMKT), a novel fine-grained mixing strategy that exploits the asynchronous nature of SNNs by interpolating RGB and DVS inputs at various time-steps. To enable label mixing in cross-modal scenarios, we further introduce modality-aware auxiliary learning objectives. These objectives support the time-step mixup process and enhance the model's ability to discriminate effectively across different modalities. Our approach enables smoother knowledge transfer, alleviates modality shift during training, and achieves superior performance in spiking image classification tasks. Extensive experiments demonstrate the effectiveness of our method across multiple datasets. The code will be released after the double-blind review process.