TrajVG: 3D Trajectory-Coupled Visual Geometry Learning
This addresses the challenge of accurate 3D reconstruction in dynamic video scenes for computer vision applications, representing a novel method for a known bottleneck.
The paper tackles the problem of feed-forward multi-frame 3D reconstruction degrading on videos with object motion by proposing TrajVG, a framework that explicitly predicts cross-frame 3D correspondence through camera-coordinate 3D trajectories, and it surpasses current feedforward performance baselines in experiments across 3D tracking, pose estimation, pointmap reconstruction, and video depth.
Feed-forward multi-frame 3D reconstruction models often degrade on videos with object motion. Global-reference becomes ambiguous under multiple motions, while the local pointmap relies heavily on estimated relative poses and can drift, causing cross-frame misalignment and duplicated structures. We propose TrajVG, a reconstruction framework that makes cross-frame 3D correspondence an explicit prediction by estimating camera-coordinate 3D trajectories. We couple sparse trajectories, per-frame local point maps, and relative camera poses with geometric consistency objectives: (i) bidirectional trajectory-pointmap consistency with controlled gradient flow, and (ii) a pose consistency objective driven by static track anchors that suppresses gradients from dynamic regions. To scale training to in-the-wild videos where 3D trajectory labels are scarce, we reformulate the same coupling constraints into self-supervised objectives using only pseudo 2D tracks, enabling unified training with mixed supervision. Extensive experiments across 3D tracking, pose estimation, pointmap reconstruction, and video depth show that TrajVG surpasses the current feedforward performance baseline.