PanoContext-Former: Panoramic Total Scene Understanding with a Transformer
This addresses panoramic scene understanding for indoor environments, offering an incremental improvement by integrating multiple tasks with a transformer-based approach.
The authors tackled holistic indoor scene understanding from a single panorama by simultaneously recovering object shapes, oriented bounding boxes, and 3D room layout, outperforming previous methods in layout estimation and 3D object detection on synthetic and real-world datasets.
Panoramic image enables deeper understanding and more holistic perception of $360^\circ$ surrounding environment, which can naturally encode enriched scene context information compared to standard perspective image. Previous work has made lots of effort to solve the scene understanding task in a bottom-up form, thus each sub-task is processed separately and few correlations are explored in this procedure. In this paper, we propose a novel method using depth prior for holistic indoor scene understanding which recovers the objects' shapes, oriented bounding boxes and the 3D room layout simultaneously from a single panorama. In order to fully utilize the rich context information, we design a transformer-based context module to predict the representation and relationship among each component of the scene. In addition, we introduce a real-world dataset for scene understanding, including photo-realistic panoramas, high-fidelity depth images, accurately annotated room layouts, and oriented object bounding boxes and shapes. Experiments on the synthetic and real-world datasets demonstrate that our method outperforms previous panoramic scene understanding methods in terms of both layout estimation and 3D object detection.