SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment
This addresses a key bottleneck in developing end-to-end embodied intelligent systems by improving 3D understanding capability and reducing semantic information loss, though it appears incremental as it builds on existing paradigms with novel integration.
The paper tackles the problem of simultaneous scene understanding and 3D reconstruction from unposed images by proposing SIU3R, an alignment-free framework that unifies these tasks via pixel-aligned 3D representation and learnable queries, achieving state-of-the-art performance on both individual and combined tasks.
Simultaneous understanding and 3D reconstruction plays an important role in developing end-to-end embodied intelligent systems. To achieve this, recent approaches resort to 2D-to-3D feature alignment paradigm, which leads to limited 3D understanding capability and potential semantic information loss. In light of this, we propose SIU3R, the first alignment-free framework for generalizable simultaneous understanding and 3D reconstruction from unposed images. Specifically, SIU3R bridges reconstruction and understanding tasks via pixel-aligned 3D representation, and unifies multiple understanding (segmentation) tasks into a set of unified learnable queries, enabling native 3D understanding without the need of alignment with 2D models. To encourage collaboration between the two tasks with shared representation, we further conduct in-depth analyses of their mutual benefits, and propose two lightweight modules to facilitate their interaction. Extensive experiments demonstrate that our method achieves state-of-the-art performance not only on the individual tasks of 3D reconstruction and understanding, but also on the task of simultaneous understanding and 3D reconstruction, highlighting the advantages of our alignment-free framework and the effectiveness of the mutual benefit designs. Project page: https://insomniaaac.github.io/siu3r/