ImitDiff: Transferring Foundation-Model Priors for Distraction Robust Visuomotor Policy
This addresses robustness in robot manipulation for real-world applications, though it is incremental as it builds on existing diffusion and foundation-model techniques.
The paper tackles the problem of visuomotor imitation learning policies degrading in complex scenes with visual distractions by proposing ImitDiff, a diffusion-based policy that uses foundation-model priors to generate semantic masks and a dual-resolution pipeline, resulting in outperforming state-of-the-art methods and achieving strong generalization in zero-shot settings with novel objects and distractions.
Visuomotor imitation learning policies enable robots to efficiently acquire manipulation skills from visual demonstrations. However, as scene complexity and visual distractions increase, policies that perform well in simple settings often experience substantial performance degradation. To address this challenge, we propose ImitDiff, a diffusion-based imitation learning policy guided by fine-grained semantics within a dual-resolution workflow. Leveraging pretrained priors of vision-language foundation models, our method transforms high-level instructions into pixel-level visual semantic masks. These masks guide a dual-resolution perception pipeline that captures both global context (e.g., overall layout) from low-resolution observation and fine-grained local features (e.g., geometric details) from high-resolution observation, enabling the policy to focus on task-relevant regions. Additionally, we introduce a consistency-driven diffusion transformer action head that bridges visual semantic conditions and real-time action generation. Extensive experiments demonstrate that ImitDiff outperforms state-of-the-art vision-language manipulation frameworks, as well as visuomotor imitation learning policies, particularly under increased scene complexity and visual distractions. Notably, ImitDiff exhibits strong generalization in zero-shot settings involving novel objects and visual distractions. Furthermore, our consistency-driven action head achieves an order-of-magnitude improvement in inference speed while maintaining competitive success rates.