Perspective-Equivariant Fine-tuning for Multispectral Demosaicing without Ground Truth
This addresses the need for real-time, high-quality multispectral imaging in applications like neurosurgery and autonomous driving, offering an incremental improvement by fine-tuning pretrained models without costly ground truth.
The paper tackled the problem of multispectral demosaicing without ground truth by proposing Perspective-Equivariant Fine-tuning for Demosaicing (PEFD), which recovers fine details like blood vessels and preserves spectral fidelity, substantially outperforming recent approaches and nearing supervised performance on intraoperative and automotive datasets.
Multispectral demosaicing is crucial to reconstruct full-resolution spectral images from snapshot mosaiced measurements, enabling real-time imaging from neurosurgery to autonomous driving. Classical methods are blurry, while supervised learning requires costly ground truth (GT) obtained from slow line-scanning systems. We propose Perspective-Equivariant Fine-tuning for Demosaicing (PEFD), a framework that learns multispectral demosaicing from mosaiced measurements alone. PEFD a) exploits the projective geometry of camera-based imaging systems to leverage a richer group structure than previous demosaicing methods to recover more null-space information, and b) learns efficiently without GT by adapting pretrained foundation models designed for 1-3 channel imaging. On intraoperative and automotive datasets, PEFD recovers fine details such as blood vessels and preserves spectral fidelity, substantially outperforming recent approaches, nearing supervised performance.