EGFormer: Towards Efficient and Generalizable Multimodal Semantic Segmentation
This addresses efficiency and generalizability issues in multimodal semantic segmentation for computer vision applications, representing a novel method for a known bottleneck.
The paper tackles the problem of inefficient multimodal semantic segmentation by proposing EGFormer, which reduces parameters by up to 88% and GFLOPs by 50% while maintaining competitive performance and achieving state-of-the-art transfer results in unsupervised domain adaptation.
Recent efforts have explored multimodal semantic segmentation using various backbone architectures. However, while most methods aim to improve accuracy, their computational efficiency remains underexplored. To address this, we propose EGFormer, an efficient multimodal semantic segmentation framework that flexibly integrates an arbitrary number of modalities while significantly reducing model parameters and inference time without sacrificing performance. Our framework introduces two novel modules. First, the Any-modal Scoring Module (ASM) assigns importance scores to each modality independently, enabling dynamic ranking based on their feature maps. Second, the Modal Dropping Module (MDM) filters out less informative modalities at each stage, selectively preserving and aggregating only the most valuable features. This design allows the model to leverage useful information from all available modalities while discarding redundancy, thus ensuring high segmentation quality. In addition to efficiency, we evaluate EGFormer on a synthetic-to-real transfer task to demonstrate its generalizability. Extensive experiments show that EGFormer achieves competitive performance with up to 88 percent reduction in parameters and 50 percent fewer GFLOPs. Under unsupervised domain adaptation settings, it further achieves state-of-the-art transfer performance compared to existing methods.