Diffusing in Someone Else's Shoes: Robotic Perspective Taking with Diffusion
This addresses the problem of reducing effort in robot training by leveraging easier-to-produce third-person demonstrations, though it is incremental as it builds on existing diffusion and perspective-taking methods.
The paper tackles the challenge of enabling humanoid robots to learn from third-person demonstrations by introducing a diffusion model that generates first-person perspectives from third-person views, allowing robots to imitate actions more easily without requiring first-person demonstrations.
Humanoid robots can benefit from their similarity to the human shape by learning from humans. When humans teach other humans how to perform actions, they often demonstrate the actions, and the learning human imitates the demonstration to get an idea of how to perform the action. Being able to mentally transfer from a demonstration seen from a third-person perspective to how it should look from a first-person perspective is fundamental for this ability in humans. As this is a challenging task, it is often simplified for robots by creating demonstrations from the first-person perspective. Creating these demonstrations allows for an easier imitation but requires more effort. Therefore, we introduce a novel diffusion model that enables the robot to learn from the third-person demonstrations directly by learning to generate the first-person perspective from the third-person perspective. The model translates the size and rotations of objects and the environment between the two perspectives. This allows us to utilise the benefits of easy-to-produce third-person demonstrations and easy-to-imitate first-person demonstrations.