Look Ma, No Hands! Agent-Environment Factorization of Egocentric Videos
This work addresses a key problem in robotics by enabling better use of human videos for training robots, though it is incremental as it builds on existing factorization and diffusion methods.
The paper tackles the challenge of using egocentric videos for robotics by separating the human hand (agent) from the environment to reduce occlusion and visual mismatch, resulting in improved inpainting quality and enhanced performance in tasks like object detection and policy learning.
The analysis and use of egocentric videos for robotic tasks is made challenging by occlusion due to the hand and the visual mismatch between the human hand and a robot end-effector. In this sense, the human hand presents a nuisance. However, often hands also provide a valuable signal, e.g. the hand pose may suggest what kind of object is being held. In this work, we propose to extract a factored representation of the scene that separates the agent (human hand) and the environment. This alleviates both occlusion and mismatch while preserving the signal, thereby easing the design of models for downstream robotics tasks. At the heart of this factorization is our proposed Video Inpainting via Diffusion Model (VIDM) that leverages both a prior on real-world images (through a large-scale pre-trained diffusion model) and the appearance of the object in earlier frames of the video (through attention). Our experiments demonstrate the effectiveness of VIDM at improving inpainting quality on egocentric videos and the power of our factored representation for numerous tasks: object detection, 3D reconstruction of manipulated objects, and learning of reward functions, policies, and affordances from videos.