Variational Probabilistic Fusion Network for RGB-T Semantic Segmentation
This addresses robust segmentation in challenging lighting conditions for applications like autonomous driving or surveillance, representing a novel method for a known bottleneck.
The paper tackled the problem of sensitivity to noise, class-imbalance, and modality bias in RGB-T semantic segmentation by proposing a Variational Probabilistic Fusion Network (VPFNet), which treats fusion features as random variables and averages segmentation results over multiple samples, achieving state-of-the-art performance on MFNet and PST900 datasets.
RGB-T semantic segmentation has been widely adopted to handle hard scenes with poor lighting conditions by fusing different modality features of RGB and thermal images. Existing methods try to find an optimal fusion feature for segmentation, resulting in sensitivity to modality noise, class-imbalance, and modality bias. To overcome the problems, this paper proposes a novel Variational Probabilistic Fusion Network (VPFNet), which regards fusion features as random variables and obtains robust segmentation by averaging segmentation results under multiple samples of fusion features. The random samples generation of fusion features in VPFNet is realized by a novel Variational Feature Fusion Module (VFFM) designed based on variation attention. To further avoid class-imbalance and modality bias, we employ the weighted cross-entropy loss and introduce prior information of illumination and category to control the proposed VFFM. Experimental results on MFNet and PST900 datasets demonstrate that the proposed VPFNet can achieve state-of-the-art segmentation performance.