Robotic Scene Cloning:Advancing Zero-Shot Robotic Scene Adaptation in Manipulation via Visual Prompt Editing
This addresses the challenge of adapting robotic manipulation to new scenes without extensive on-site data collection, representing a novel method for a known bottleneck in robotics.
The paper tackles the problem of limited zero-shot capabilities in deploying pre-trained robot models to real-world user scenarios by proposing Robotic Scene Cloning (RSC), a method for scene-specific adaptation via visual prompt editing of existing trajectories, which significantly enhances policy generalization in target environments as demonstrated in experiments across simulated and real-world settings.
Modern robots can perform a wide range of simple tasks and adapt to diverse scenarios in the well-trained environment. However, deploying pre-trained robot models in real-world user scenarios remains challenging due to their limited zero-shot capabilities, often necessitating extensive on-site data collection. To address this issue, we propose Robotic Scene Cloning (RSC), a novel method designed for scene-specific adaptation by editing existing robot operation trajectories. RSC achieves accurate and scene-consistent sample generation by leveraging a visual prompting mechanism and a carefully tuned condition injection module. Not only transferring textures but also performing moderate shape adaptations in response to the visual prompts, RSC demonstrates reliable task performance across a variety of object types. Experiments across various simulated and real-world environments demonstrate that RSC significantly enhances policy generalization in target environments.