SPKLIP: Aligning Spike Video Streams with Natural Language
This work addresses the challenge of semantic understanding in spike cameras for event-based multimodal research, with potential applications in neuromorphic deployment.
The paper tackles the problem of aligning spike video streams with natural language, where existing models like CLIP underperform due to modality mismatch, and introduces SPKLIP, which achieves state-of-the-art performance on benchmark datasets and demonstrates strong few-shot generalization on a new real-world dataset.
Spike cameras offer unique sensing capabilities but their sparse, asynchronous output challenges semantic understanding, especially for Spike Video-Language Alignment (Spike-VLA) where models like CLIP underperform due to modality mismatch. We introduce SPKLIP, the first architecture specifically for Spike-VLA. SPKLIP employs a hierarchical spike feature extractor that adaptively models multi-scale temporal dynamics in event streams, and uses spike-text contrastive learning to directly align spike video with language, enabling effective few-shot learning. A full-spiking visual encoder variant, integrating SNN components into our pipeline, demonstrates enhanced energy efficiency. Experiments show state-of-the-art performance on benchmark spike datasets and strong few-shot generalization on a newly contributed real-world dataset. SPKLIP's energy efficiency highlights its potential for neuromorphic deployment, advancing event-based multimodal research. The source code and dataset are available at [link removed for anonymity].