PlaneFormers: From Sparse View Planes to 3D Reconstruction
This addresses the challenge of 3D reconstruction from sparse views for applications in computer vision, though it appears incremental as it builds on transformer architectures with domain-specific modifications.
The paper tackles the problem of planar surface reconstruction from images with limited overlap by introducing PlaneFormer, a transformer-based method that uses 3D-aware plane tokens for 3D reasoning, achieving substantially more effective results than prior work.
We present an approach for the planar surface reconstruction of a scene from images with limited overlap. This reconstruction task is challenging since it requires jointly reasoning about single image 3D reconstruction, correspondence between images, and the relative camera pose between images. Past work has proposed optimization-based approaches. We introduce a simpler approach, the PlaneFormer, that uses a transformer applied to 3D-aware plane tokens to perform 3D reasoning. Our experiments show that our approach is substantially more effective than prior work, and that several 3D-specific design decisions are crucial for its success.