Diffusion-based RGB-D Semantic Segmentation with Deformable Attention Transformer
This work addresses scene understanding for autonomous systems by improving segmentation accuracy in noisy RGB-D data, representing an incremental advance with specific gains in robustness and efficiency.
The authors tackled RGB-D semantic segmentation by introducing a diffusion-based framework with a Deformable Attention Transformer encoder, achieving state-of-the-art performance on NYUv2 and SUN-RGBD datasets with robust results in challenging scenarios and reduced training time.
Vision-based perception and reasoning is essential for scene understanding in any autonomous system. RGB and depth images are commonly used to capture both the semantic and geometric features of the environment. Developing methods to reliably interpret this data is critical for real-world applications, where noisy measurements are often unavoidable. In this work, we introduce a diffusion-based framework to address the RGB-D semantic segmentation problem. Additionally, we demonstrate that utilizing a Deformable Attention Transformer as the encoder to extract features from depth images effectively captures the characteristics of invalid regions in depth measurements. Our generative framework shows a greater capacity to model the underlying distribution of RGB-D images, achieving robust performance in challenging scenarios with significantly less training time compared to discriminative methods. Experimental results indicate that our approach achieves State-of-the-Art performance on both the NYUv2 and SUN-RGBD datasets in general and especially in the most challenging of their image data. Our project page will be available at https://diffusionmms.github.io/