PreF3R: Pose-Free Feed-Forward 3D Gaussian Splatting from Variable-length Image Sequence
This addresses the challenge of pose-free novel-view synthesis for applications requiring efficient 3D reconstruction from variable-length image sequences, representing a novel method for a known bottleneck.
The paper tackles the problem of 3D reconstruction from unposed image sequences by introducing PreF3R, which eliminates the need for camera calibration and reconstructs a 3D Gaussian field directly at 20 FPS, enabling real-time novel-view rendering with robust generalization to unseen scenes.
We present PreF3R, Pose-Free Feed-forward 3D Reconstruction from an image sequence of variable length. Unlike previous approaches, PreF3R removes the need for camera calibration and reconstructs the 3D Gaussian field within a canonical coordinate frame directly from a sequence of unposed images, enabling efficient novel-view rendering. We leverage DUSt3R's ability for pair-wise 3D structure reconstruction, and extend it to sequential multi-view input via a spatial memory network, eliminating the need for optimization-based global alignment. Additionally, PreF3R incorporates a dense Gaussian parameter prediction head, which enables subsequent novel-view synthesis with differentiable rasterization. This allows supervising our model with the combination of photometric loss and pointmap regression loss, enhancing both photorealism and structural accuracy. Given a sequence of ordered images, PreF3R incrementally reconstructs the 3D Gaussian field at 20 FPS, therefore enabling real-time novel-view rendering. Empirical experiments demonstrate that PreF3R is an effective solution for the challenging task of pose-free feed-forward novel-view synthesis, while also exhibiting robust generalization to unseen scenes.