Online panoptic 3D reconstruction as a Linear Assignment Problem
This work addresses the need for machines to interpret their surroundings in 3D for real-time holistic scene understanding, though it is incremental as it builds on existing panoptic segmentation methods.
The paper tackles the problem of real-time 3D reconstruction from panoptic image segmentations by developing an online algorithm that processes data sequentially, outperforming earlier similar works in online operation and achieving high frame rates suitable for real-time applications.
Real-time holistic scene understanding would allow machines to interpret their surrounding in a much more detailed manner than is currently possible. While panoptic image segmentation methods have brought image segmentation closer to this goal, this information has to be described relative to the 3D environment for the machine to be able to utilise it effectively. In this paper, we investigate methods for sequentially reconstructing static environments from panoptic image segmentations in 3D. We specifically target real-time operation: the algorithm must process data strictly online and be able to run at relatively fast frame rates. Additionally, the method should be scalable for environments large enough for practical applications. By applying a simple but powerful data-association algorithm, we outperform earlier similar works when operating purely online. Our method is also capable of reaching frame-rates high enough for real-time applications and is scalable to larger environments as well. Source code and further demonstrations are released to the public at: \url{https://tutvision.github.io/Online-Panoptic-3D/}