Multispectral and Hyperspectral Image Fusion by MS/HS Fusion Net
This work addresses a practical limitation in hyperspectral imaging for applications like material analysis, but it is incremental as it builds on existing fusion methods with a novel network design.
The paper tackles the problem of generating high-resolution hyperspectral images from high-resolution multispectral and low-resolution hyperspectral inputs, using a model-based deep learning approach called MS/HS Fusion Net, and demonstrates superior performance compared to state-of-the-art methods in experiments on simulated and real data.
Hyperspectral imaging can help better understand the characteristics of different materials, compared with traditional image systems. However, only high-resolution multispectral (HrMS) and low-resolution hyperspectral (LrHS) images can generally be captured at video rate in practice. In this paper, we propose a model-based deep learning approach for merging an HrMS and LrHS images to generate a high-resolution hyperspectral (HrHS) image. In specific, we construct a novel MS/HS fusion model which takes the observation models of low-resolution images and the low-rankness knowledge along the spectral mode of HrHS image into consideration. Then we design an iterative algorithm to solve the model by exploiting the proximal gradient method. And then, by unfolding the designed algorithm, we construct a deep network, called MS/HS Fusion Net, with learning the proximal operators and model parameters by convolutional neural networks. Experimental results on simulated and real data substantiate the superiority of our method both visually and quantitatively as compared with state-of-the-art methods along this line of research.