CrossWeaver: Cross-modal Weaving for Arbitrary-Modality Semantic Segmentation
This work addresses the challenge of arbitrary-modality semantic segmentation for applications like autonomous driving or robotics, representing an incremental improvement over existing fusion strategies.
The paper tackles the problem of limited flexibility and ineffective cross-modal coordination in multimodal semantic segmentation by proposing CrossWeaver, a framework that achieves state-of-the-art performance with minimal additional parameters and strong generalization to unseen modality combinations.
Multimodal semantic segmentation has shown great potential in leveraging complementary information across diverse sensing modalities. However, existing approaches often rely on carefully designed fusion strategies that either use modality-specific adaptations or rely on loosely coupled interactions, thereby limiting flexibility and resulting in less effective cross-modal coordination. Moreover, these methods often struggle to balance efficient information exchange with preserving the unique characteristics of each modality across different modality combinations. To address these challenges, we propose CrossWeaver, a simple yet effective multimodal fusion framework for arbitrary-modality semantic segmentation. Its core is a Modality Interaction Block (MIB), which enables selective and reliability-aware cross-modal interaction within the encoder, while a lightweight Seam-Aligned Fusion (SAF) module further aggregates the enhanced features. Extensive experiments on multiple multimodal semantic segmentation benchmarks demonstrate that our framework achieves state-of-the-art performance with minimal additional parameters and strong generalization to unseen modality combinations.