SCALMU: Synthetically-trained Coupling of Adaptive Learned Multiplicative Updates for Hyperspectral-Multispectral Fusion
This work provides an improved method for hyperspectral-multispectral image fusion, benefiting applications requiring high-resolution spectral imaging, such as remote sensing and medical imaging, by offering better performance while retaining physical interpretability.
This paper addresses the challenge of fusing hyperspectral and multispectral images to achieve high-resolution hyperspectral imaging. The authors propose SCALMU, an unrolled neural network that integrates adaptive learnable matrices into the classical Coupled Nonnegative Matrix Factorization framework, demonstrating superior results over state-of-the-art methods on several datasets.
HyperSpectral-MultiSpectral Image (HSI-MSI) fusion enables high-resolution hyperspectral imaging by combining the rich spectral information of low-spatial-resolution hyperspectral images with the detailed spatial structure of multispectral images. Classical methods such as Coupled Nonnegative Matrix Factorization (CNMF) benefit from a strong physical interpretability but suffer from inferior results compared to their deep-learning counterparts. To address this limitation, we propose SCALMU (Synthetically-trained Coupling of Adaptive Learned Multiplicative Updates), a novel unrolled neural network architecture that integrates adaptive learnable matrices within the classical framework of CNMF multiplicative updates, improving its results. Due to its architectural proximity with CNMF, the resulting algorithm preserves physical interpretability and nonnegativity constraints. To overcome data scarcity for training, we additionally generate a synthetic HSI-MSI dataset via the dead leaves model, enabling synthetic supervision. SCALMU is then trained end-to-end on this dataset. Experiments demonstrate SCALMU's superiority over state-of-the-art methods on several datasets. The code is available at https://github.com/xinxinxu99/SCALMU.git