PlaneRecTR++: Unified Query Learning for Joint 3D Planar Reconstruction and Pose Estimation
This addresses the challenge of fragmented multi-view 3D reconstruction for computer vision applications, offering a unified approach that improves performance.
The paper tackles the problem of 3D planar reconstruction from images, which involves multiple sub-tasks like plane detection and camera pose estimation, by proposing PlaneRecTR++, a Transformer-based architecture that unifies these tasks in a single-stage framework, achieving state-of-the-art performance on datasets including ScanNetv1, ScanNetv2, NYUv2-Plane, and MatterPort3D.
The challenging task of 3D planar reconstruction from images involves several sub-tasks including frame-wise plane detection, segmentation, parameter regression and possibly depth prediction, along with cross-frame plane correspondence and relative camera pose estimation. Previous works adopt a divide and conquer strategy, addressing above sub-tasks with distinct network modules in a two-stage paradigm. Specifically, given an initial camera pose and per-frame plane predictions from the first stage, further exclusively designed modules relying on external plane correspondence labeling are applied to merge multi-view plane entities and produce refined camera pose. Notably, existing work fails to integrate these closely related sub-tasks into a unified framework, and instead addresses them separately and sequentially, which we identify as a primary source of performance limitations. Motivated by this finding and the success of query-based learning in enriching reasoning among semantic entities, in this paper, we propose PlaneRecTR++, a Transformer-based architecture, which for the first time unifies all tasks of multi-view planar reconstruction and pose estimation within a compact single-stage framework, eliminating the need for the initial pose estimation and supervision of plane correspondence. Extensive quantitative and qualitative experiments demonstrate that our proposed unified learning achieves mutual benefits across sub-tasks, achieving a new state-of-the-art performance on the public ScanNetv1, ScanNetv2, NYUv2-Plane, and MatterPort3D datasets. Codes are available at https://github.com/SJingjia/PlaneRecTR-PP.