DemosaicFormer: Coarse-to-Fine Demosaicing Network for HybridEVS Camera
This addresses a specific problem for smartphone and wearable device applications requiring low-light, high dynamic range, and low-latency imaging, but it is incremental as it builds on existing demosaicing methods for a new sensor type.
The paper tackles the lack of an image signal processing pipeline for Hybrid Event-Based Vision Sensors by proposing DemosaicFormer, a coarse-to-fine demosaicing network, which achieves the best performance in the MIPI 2024 challenge on all evaluation metrics.
Hybrid Event-Based Vision Sensor (HybridEVS) is a novel sensor integrating traditional frame-based and event-based sensors, offering substantial benefits for applications requiring low-light, high dynamic range, and low-latency environments, such as smartphones and wearable devices. Despite its potential, the lack of Image signal processing (ISP) pipeline specifically designed for HybridEVS poses a significant challenge. To address this challenge, in this study, we propose a coarse-to-fine framework named DemosaicFormer which comprises coarse demosaicing and pixel correction. Coarse demosaicing network is designed to produce a preliminary high-quality estimate of the RGB image from the HybridEVS raw data while the pixel correction network enhances the performance of image restoration and mitigates the impact of defective pixels. Our key innovation is the design of a Multi-Scale Gating Module (MSGM) applying the integration of cross-scale features, which allows feature information to flow between different scales. Additionally, the adoption of progressive training and data augmentation strategies further improves model's robustness and effectiveness. Experimental results show superior performance against the existing methods both qualitatively and visually, and our DemosaicFormer achieves the best performance in terms of all the evaluation metrics in the MIPI 2024 challenge on Demosaic for Hybridevs Camera. The code is available at https://github.com/QUEAHREN/DemosaicFormer.