Interp3R: Continuous-time 3D Geometry Estimation with Frames and Events
This addresses the limitation of blind time between frames in 3D geometry estimation for applications like robotics and AR/VR, though it is an incremental enhancement of existing pointmap-based models.
The paper tackles the problem of estimating 3D geometry only at discrete time instants in pointmap-based models by introducing Interp3R, which uses event data to interpolate pointmaps for continuous-time depth and camera pose estimation, outperforming state-of-the-art baselines by a considerable margin.
In recent years, 3D visual foundation models pioneered by pointmap-based approaches such as DUSt3R have attracted a lot of interest, achieving impressive accuracy and strong generalization across diverse scenes. However, these methods are inherently limited to recovering scene geometry only at the discrete time instants when images are captured, leaving the scene evolution during the blind time between consecutive frames largely unexplored. We introduce Interp3R, to the best of our knowledge the first method that enhances pointmap-based models to estimate depth and camera poses at arbitrary time instants. Interp3R leverages asynchronous event data to interpolate pointmaps produced by frame-based models, enabling temporally continuous geometric representations. Depth and camera poses are then jointly recovered by aligning the interpolated pointmaps together with those predicted by the underlying frame-based models into a consistent spatial framework. We train Interp3R exclusively on a synthetic dataset, yet demonstrate strong generalization across a wide range of synthetic and real-world benchmarks. Extensive experiments show that Interp3R outperforms by a considerable margin state-of-the-art baselines that follow a two-stage pipeline of 2D video frame interpolation followed by 3D geometry estimation.