Learning Social Affordance Grammar from Videos: Transferring Human Interactions to Human-Robot Interactions
This work addresses the challenge of transferring human interaction patterns to robots for improved human-robot interaction, representing an incremental advance in robotics and AI.
The paper tackles the problem of enabling human-robot interactions by learning social affordance grammar from RGB-D videos of human interactions, resulting in a model that generates human-like behaviors in unseen scenarios and outperforms baselines in experiments including Baxter simulation and real tests.
In this paper, we present a general framework for learning social affordance grammar as a spatiotemporal AND-OR graph (ST-AOG) from RGB-D videos of human interactions, and transfer the grammar to humanoids to enable a real-time motion inference for human-robot interaction (HRI). Based on Gibbs sampling, our weakly supervised grammar learning can automatically construct a hierarchical representation of an interaction with long-term joint sub-tasks of both agents and short term atomic actions of individual agents. Based on a new RGB-D video dataset with rich instances of human interactions, our experiments of Baxter simulation, human evaluation, and real Baxter test demonstrate that the model learned from limited training data successfully generates human-like behaviors in unseen scenarios and outperforms both baselines.