Orientation-aware Semantic Segmentation on Icosahedron Spheres
This addresses the problem of efficient high-resolution semantic segmentation on spherical images for applications like autonomous driving, though it appears incremental as it builds on existing icosahedron mesh methods.
The paper tackles semantic segmentation on omnidirectional images by proposing an orientation-aware CNN framework for icosahedron meshes, achieving new state-of-the-art results on the 2D3DS dataset and Omni-SYNTHIA version of SYNTHIA while enabling memory-efficient execution up to level-8 resolution meshes (equivalent to 640 x 1024 equirectangular images).
We address semantic segmentation on omnidirectional images, to leverage a holistic understanding of the surrounding scene for applications like autonomous driving systems. For the spherical domain, several methods recently adopt an icosahedron mesh, but systems are typically rotation invariant or require significant memory and parameters, thus enabling execution only at very low resolutions. In our work, we propose an orientation-aware CNN framework for the icosahedron mesh. Our representation allows for fast network operations, as our design simplifies to standard network operations of classical CNNs, but under consideration of north-aligned kernel convolutions for features on the sphere. We implement our representation and demonstrate its memory efficiency up-to a level-8 resolution mesh (equivalent to 640 x 1024 equirectangular images). Finally, since our kernels operate on the tangent of the sphere, standard feature weights, pretrained on perspective data, can be directly transferred with only small need for weight refinement. In our evaluation our orientation-aware CNN becomes a new state of the art for the recent 2D3DS dataset, and our Omni-SYNTHIA version of SYNTHIA. Rotation invariant classification and segmentation tasks are additionally presented for comparison to prior art.