MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusion
This work addresses the problem of fine-grained image fusion for applications like low-light object detection and medical imaging, offering a novel approach but likely incremental in the broader field of multimodal fusion.
The paper tackles the problem of infrared and visible image fusion by addressing the limitation of existing methods in capturing high-level semantic information, introducing MSGFusion, a framework that uses multimodal scene graphs to guide fusion, resulting in significant performance improvements over state-of-the-art approaches in detail preservation and structural clarity.
Infrared and visible image fusion has garnered considerable attention owing to the strong complementarity of these two modalities in complex, harsh environments. While deep learning-based fusion methods have made remarkable advances in feature extraction, alignment, fusion, and reconstruction, they still depend largely on low-level visual cues, such as texture and contrast, and struggle to capture the high-level semantic information embedded in images. Recent attempts to incorporate text as a source of semantic guidance have relied on unstructured descriptions that neither explicitly model entities, attributes, and relationships nor provide spatial localization, thereby limiting fine-grained fusion performance. To overcome these challenges, we introduce MSGFusion, a multimodal scene graph-guided fusion framework for infrared and visible imagery. By deeply coupling structured scene graphs derived from text and vision, MSGFusion explicitly represents entities, attributes, and spatial relations, and then synchronously refines high-level semantics and low-level details through successive modules for scene graph representation, hierarchical aggregation, and graph-driven fusion. Extensive experiments on multiple public benchmarks show that MSGFusion significantly outperforms state-of-the-art approaches, particularly in detail preservation and structural clarity, and delivers superior semantic consistency and generalizability in downstream tasks such as low-light object detection, semantic segmentation, and medical image fusion.