VGGT-DP: Generalizable Robot Control via Vision Foundation Models
This addresses the problem of robust robot manipulation for robotics researchers, though it appears incremental as it builds on existing visual imitation learning methods.
The paper tackles the problem of limited spatial understanding and generalization in visual imitation learning for robot control by proposing VGGT-DP, a framework that integrates geometric priors from a pretrained 3D perception model with proprioceptive feedback. The result is significant performance improvements over baselines like DP and DP3 on MetaWorld tasks, especially in precision-critical and long-horizon scenarios.
Visual imitation learning frameworks allow robots to learn manipulation skills from expert demonstrations. While existing approaches mainly focus on policy design, they often neglect the structure and capacity of visual encoders, limiting spatial understanding and generalization. Inspired by biological vision systems, which rely on both visual and proprioceptive cues for robust control, we propose VGGT-DP, a visuomotor policy framework that integrates geometric priors from a pretrained 3D perception model with proprioceptive feedback. We adopt the Visual Geometry Grounded Transformer (VGGT) as the visual encoder and introduce a proprioception-guided visual learning strategy to align perception with internal robot states, improving spatial grounding and closed-loop control. To reduce inference latency, we design a frame-wise token reuse mechanism that compacts multi-view tokens into an efficient spatial representation. We further apply random token pruning to enhance policy robustness and reduce overfitting. Experiments on challenging MetaWorld tasks show that VGGT-DP significantly outperforms strong baselines such as DP and DP3, particularly in precision-critical and long-horizon scenarios.