Interpretable Deep Multimodal Image Super-Resolution
This work addresses the problem of improving image super-resolution quality for applications like remote sensing or medical imaging by effectively fusing information from another modality, though it is incremental as it builds on existing deep multimodal models.
The paper tackles multimodal image super-resolution by integrating coupled sparse priors into a deep network design, resulting in a model that outperforms state-of-the-art methods for near-infrared image super-resolution guided by RGB images.
Multimodal image super-resolution (SR) is the reconstruction of a high resolution image given a low-resolution observation with the aid of another image modality. While existing deep multimodal models do not incorporate domain knowledge about image SR, we present a multimodal deep network design that integrates coupled sparse priors and allows the effective fusion of information from another modality into the reconstruction process. Our method is inspired by a novel iterative algorithm for coupled convolutional sparse coding, resulting in an interpretable network by design. We apply our model to the super-resolution of near-infrared image guided by RGB images. Experimental results show that our model outperforms state-of-the-art methods.