TranSplat: Generalizable 3D Gaussian Splatting from Sparse Multi-View Images with Transformers
This work addresses a key bottleneck in efficient 3D reconstruction from sparse images, offering improvements for applications like robotics and AR/VR, though it is incremental over existing G-3DGS methods.
The paper tackles the challenge of accurate multi-view feature matching in Generalizable 3D Gaussian Splatting for sparse-view 3D reconstruction by using a depth confidence map and monocular depth priors, achieving state-of-the-art performance on RealEstate10K and ACID benchmarks with competitive speed and strong generalization.
Compared with previous 3D reconstruction methods like Nerf, recent Generalizable 3D Gaussian Splatting (G-3DGS) methods demonstrate impressive efficiency even in the sparse-view setting. However, the promising reconstruction performance of existing G-3DGS methods relies heavily on accurate multi-view feature matching, which is quite challenging. Especially for the scenes that have many non-overlapping areas between various views and contain numerous similar regions, the matching performance of existing methods is poor and the reconstruction precision is limited. To address this problem, we develop a strategy that utilizes a predicted depth confidence map to guide accurate local feature matching. In addition, we propose to utilize the knowledge of existing monocular depth estimation models as prior to boost the depth estimation precision in non-overlapping areas between views. Combining the proposed strategies, we present a novel G-3DGS method named TranSplat, which obtains the best performance on both the RealEstate10K and ACID benchmarks while maintaining competitive speed and presenting strong cross-dataset generalization ability. Our code, and demos will be available at: https://xingyoujun.github.io/transplat.