REL-SF4PASS: Panoramic Semantic Segmentation with REL Depth Representation and Spherical Fusion
This work addresses the problem of incomplete scene perception in panoramic images for computer vision applications, representing an incremental improvement over existing methods.
The paper tackles panoramic semantic segmentation by proposing a new depth representation (REL) and spherical fusion method (SMMF) to better utilize panoramic geometry, resulting in a 2.35% average mIoU improvement and 70% reduction in performance variance under 3D disturbance on the Stanford2D3D dataset.
As an important and challenging problem in computer vision, Panoramic Semantic Segmentation (PASS) aims to give complete scene perception based on an ultra-wide angle of view. Most PASS methods often focus on spherical geometry with RGB input or using the depth information in original or HHA format, which does not make full use of panoramic image geometry. To address these shortcomings, we propose REL-SF4PASS with our REL depth representation based on cylindrical coordinate and Spherical-dynamic Multi-Modal Fusion SMMF. REL is made up of Rectified Depth, Elevation-Gained Vertical Inclination Angle, and Lateral Orientation Angle, which fully represents 3D space in cylindrical coordinate style and the surface normal direction. SMMF aims to ensure the diversity of fusion for different panoramic image regions and reduce the breakage of cylinder side surface expansion in ERP projection, which uses different fusion strategies to match the different regions in panoramic images. Experimental results show that REL-SF4PASS considerably improves performance and robustness on popular benchmark, Stanford2D3D Panoramic datasets. It gains 2.35% average mIoU improvement on all 3 folds and reduces the performance variance by approximately 70% when facing 3D disturbance.