SegRGB-X: General RGB-X Semantic Segmentation Model
This addresses the problem of redundant development efforts for researchers and practitioners in computer vision by providing a unified model for arbitrary-modal segmentation.
The paper tackles the challenge of semantic segmentation across diverse sensor modalities by introducing a universal framework, achieving state-of-the-art performance with a mIoU of 65.03% on five datasets.
Semantic segmentation across arbitrary sensor modalities faces significant challenges due to diverse sensor characteristics, and the traditional configurations for this task result in redundant development efforts. We address these challenges by introducing a universal arbitrary-modal semantic segmentation framework that unifies segmentation across multiple modalities. Our approach features three key innovations: (1) the Modality-aware CLIP (MA-CLIP), which provides modality-specific scene understanding guidance through LoRA fine-tuning; (2) Modality-aligned Embeddings for capturing fine-grained features; and (3) the Domain-specific Refinement Module (DSRM) for dynamic feature adjustment. Evaluated on five diverse datasets with different complementary modalities (event, thermal, depth, polarization, and light field), our model surpasses specialized multi-modal methods and achieves state-of-the-art performance with a mIoU of 65.03%. The codes will be released upon acceptance.