Rethinking High-speed Image Reconstruction Framework with Spike Camera
This work addresses the problem of noisy image reconstruction for spike cameras in low-light environments, offering a novel approach that could benefit high-speed imaging applications, though it appears incremental as it builds on existing CLIP models.
The paper tackles the challenge of reconstructing high-quality images from spike camera data under low-light conditions, introducing SpikeCLIP, which uses CLIP and textual descriptions to improve texture details and luminance balance, achieving significant enhancements on real-world datasets like U-CALTECH and U-CIFAR.
Spike cameras, as innovative neuromorphic devices, generate continuous spike streams to capture high-speed scenes with lower bandwidth and higher dynamic range than traditional RGB cameras. However, reconstructing high-quality images from the spike input under low-light conditions remains challenging. Conventional learning-based methods often rely on the synthetic dataset as the supervision for training. Still, these approaches falter when dealing with noisy spikes fired under the low-light environment, leading to further performance degradation in the real-world dataset. This phenomenon is primarily due to inadequate noise modelling and the domain gap between synthetic and real datasets, resulting in recovered images with unclear textures, excessive noise, and diminished brightness. To address these challenges, we introduce a novel spike-to-image reconstruction framework SpikeCLIP that goes beyond traditional training paradigms. Leveraging the CLIP model's powerful capability to align text and images, we incorporate the textual description of the captured scene and unpaired high-quality datasets as the supervision. Our experiments on real-world low-light datasets U-CALTECH and U-CIFAR demonstrate that SpikeCLIP significantly enhances texture details and the luminance balance of recovered images. Furthermore, the reconstructed images are well-aligned with the broader visual features needed for downstream tasks, ensuring more robust and versatile performance in challenging environments.