Beyond Calibration: Physically Informed Learning for Raw-to-Raw Mapping
This addresses the challenge of raw-to-raw mapping for seamless image processing in devices with varying sensors and optics, representing an incremental improvement over existing methods.
The paper tackled the problem of achieving consistent color reproduction across multiple cameras for image fusion and ISP compatibility, introducing the Neural Physical Model (NPM) which outperformed state-of-the-art methods on datasets like NUS and BeyondRGB.
Achieving consistent color reproduction across multiple cameras is essential for seamless image fusion and Image Processing Pipeline (ISP) compatibility in modern devices, but it is a challenging task due to variations in sensors and optics. Existing raw-to-raw conversion methods face limitations such as poor adaptability to changing illumination, high computational costs, or impractical requirements such as simultaneous camera operation and overlapping fields-of-view. We introduce the Neural Physical Model (NPM), a lightweight, physically-informed approach that simulates raw images under specified illumination to estimate transformations between devices. The NPM effectively adapts to varying illumination conditions, can be initialized with physical measurements, and supports training with or without paired data. Experiments on public datasets like NUS and BeyondRGB demonstrate that NPM outperforms recent state-of-the-art methods, providing robust chromatic consistency across different sensors and optical systems.