Automated acquisition of structured, semantic models of manipulation activities from human VR demonstration
This system addresses the problem of efficiently acquiring robot-understandable models of human manipulation activities for robotics researchers, potentially reducing the effort in programming robots for everyday tasks.
This paper introduces a system for collecting and annotating human manipulation activities demonstrated in a virtual reality environment. The system records full-body movements and eye-tracking data, along with physical interactions, to generate structured, semantic models of these activities.
In this paper we present a system capable of collecting and annotating, human performed, robot understandable, everyday activities from virtual environments. The human movements are mapped in the simulated world using off-the-shelf virtual reality devices with full body, and eye tracking capabilities. All the interactions in the virtual world are physically simulated, thus movements and their effects are closely relatable to the real world. During the activity execution, a subsymbolic data logger is recording the environment and the human gaze on a per-frame basis, enabling offline scene reproduction and replays. Coupled with the physics engine, online monitors (symbolic data loggers) are parsing (using various grammars) and recording events, actions, and their effects in the simulated world.