Modulate and Reconstruct: Learning Hyperspectral Imaging from Misaligned Smartphone Views
This addresses the problem of limited accuracy in hyperspectral imaging for practical applications using commodity hardware, representing a novel method for a known bottleneck.
The paper tackles hyperspectral reconstruction from RGB images by proposing a multi-image framework using a triple-camera smartphone with spectral filters, achieving 30% more accurate spectra compared to ordinary RGB cameras.
Hyperspectral reconstruction (HSR) from RGB images is a fundamentally ill-posed problem due to severe spectral information loss. Existing approaches typically rely on a single RGB image, limiting reconstruction accuracy. In this work, we propose a novel multi-image-to-hyperspectral reconstruction (MI-HSR) framework that leverages a triple-camera smartphone system, where two lenses are equipped with carefully selected spectral filters. Our configuration, grounded in theoretical and empirical analysis, enables richer and more diverse spectral observations than conventional single-camera setups. To support this new paradigm, we introduce Doomer, the first dataset for MI-HSR, comprising aligned images from three smartphone cameras and a hyperspectral reference camera across diverse scenes. We show that the proposed HSR model achieves consistent improvements over existing methods on the newly proposed benchmark. In a nutshell, our setup allows 30% towards more accurately estimated spectra compared to an ordinary RGB camera. Our findings suggest that multi-view spectral filtering with commodity hardware can unlock more accurate and practical hyperspectral imaging solutions.