MCGA: Mixture of Codebooks Hyperspectral Reconstruction via Grayscale-Aware Attention
This addresses the cost and computational limitations of hyperspectral imaging for applications like remote sensing or medical imaging, though it is an incremental improvement over existing methods.
The paper tackles the ill-posed problem of reconstructing hyperspectral images from RGB inputs by proposing MCGA, which uses spectral priors and photometric consistency to achieve state-of-the-art accuracy, strong cross-dataset generalization, and 4-5x faster inference.
Reconstructing hyperspectral images (HSIs) from RGB inputs provides a cost-effective alternative to hyperspectral cameras, but reconstructing high-dimensional spectra from three channels is inherently ill-posed. Existing methods typically directly regress RGB-to-HSI mappings using large attention networks, which are computationally expensive and handle ill-posedness only implicitly. We propose MCGA, a Mixture-of-Codebooks with Grayscale-aware Attention framework that explicitly addresses these challenges using spectral priors and photometric consistency. MCGA first learns transferable spectral priors via a mixture-of-codebooks (MoC) from heterogeneous HSI datasets, then aligns RGB features with these priors through grayscale-aware photometric attention (GANet). Efficiency and robustness are further improved via top-K attention design and test-time adaptation (TTA). Experiments on benchmarks and real-world data demonstrate the state-of-the-art accuracy, strong cross-dataset generalization, and 4-5x faster inference. Codes will be available once acceptance at https://github.com/Fibonaccirabbit/MCGA.