Fusing in 3D: Free-Viewpoint Fusion Rendering with a 3D Infrared-Visible Scene Representation
This addresses the limitation of existing 2D fusion methods that lose critical scene information by providing a more comprehensive 3D approach for applications like surveillance or autonomous systems, though it is incremental as it builds on prior fusion and 3D representation techniques.
The paper tackles the problem of infrared-visible image fusion by proposing a 3D scene representation framework that reconstructs geometry from multimodal inputs and enables free-viewpoint rendering, achieving improved performance as demonstrated through experiments.
Infrared-visible image fusion aims to integrate infrared and visible information into a single fused image. Existing 2D fusion methods focus on fusing images from fixed camera viewpoints, neglecting a comprehensive understanding of complex scenarios, which results in the loss of critical information about the scene. To address this limitation, we propose a novel Infrared-Visible Gaussian Fusion (IVGF) framework, which reconstructs scene geometry from multimodal 2D inputs and enables direct rendering of fused images. Specifically, we propose a cross-modal adjustment (CMA) module that modulates the opacity of Gaussians to solve the problem of cross-modal conflicts. Moreover, to preserve the distinctive features from both modalities, we introduce a fusion loss that guides the optimization of CMA, thus ensuring that the fused image retains the critical characteristics of each modality. Comprehensive qualitative and quantitative experiments demonstrate the effectiveness of the proposed method.