Improved Mapping Between Illuminations and Sensors for RAW Images
This work addresses the problem of reducing data capture burden for RAW image processing in computer vision, particularly for camera manufacturers and researchers, but it is incremental as it builds on existing mapping methods with a new dataset.
The paper tackles the challenge of capturing RAW image datasets for deep learning by introducing a novel dataset with 390 illuminations, four cameras, and 18 scenes, and proposes a lightweight neural network for illumination and sensor mapping that outperforms existing methods, demonstrating its utility in training neural ISPs.
RAW images are unprocessed camera sensor output with sensor-specific RGB values based on the sensor's color filter spectral sensitivities. RAW images also incur strong color casts due to the sensor's response to the spectral properties of scene illumination. The sensor- and illumination-specific nature of RAW images makes it challenging to capture RAW datasets for deep learning methods, as scenes need to be captured for each sensor and under a wide range of illumination. Methods for illumination augmentation for a given sensor and the ability to map RAW images between sensors are important for reducing the burden of data capture. To explore this problem, we introduce the first-of-its-kind dataset comprising carefully captured scenes under a wide range of illumination. Specifically, we use a customized lightbox with tunable illumination spectra to capture several scenes with different cameras. Our illumination and sensor mapping dataset has 390 illuminations, four cameras, and 18 scenes. Using this dataset, we introduce a lightweight neural network approach for illumination and sensor mapping that outperforms competing methods. We demonstrate the utility of our approach on the downstream task of training a neural ISP. Link to project page: https://github.com/SamsungLabs/illum-sensor-mapping.