Learning the Semantics of Manipulation Action
This work addresses the challenge of understanding and executing manipulation actions for applications in robotics and human action observation, representing an incremental advancement in semantic modeling.
The paper tackles the problem of modeling manipulation actions by introducing a formal computational framework based on Combinatory Categorial Grammar, which enables semantic parsing from video data and reasoning beyond observations, with validation on a large public dataset.
In this paper we present a formal computational framework for modeling manipulation actions. The introduced formalism leads to semantics of manipulation action and has applications to both observing and understanding human manipulation actions as well as executing them with a robotic mechanism (e.g. a humanoid robot). It is based on a Combinatory Categorial Grammar. The goal of the introduced framework is to: (1) represent manipulation actions with both syntax and semantic parts, where the semantic part employs $λ$-calculus; (2) enable a probabilistic semantic parsing schema to learn the $λ$-calculus representation of manipulation action from an annotated action corpus of videos; (3) use (1) and (2) to develop a system that visually observes manipulation actions and understands their meaning while it can reason beyond observations using propositional logic and axiom schemata. The experiments conducted on a public available large manipulation action dataset validate the theoretical framework and our implementation.