Motion Question Answering via Modular Motion Programs
This work addresses the challenge of building AI systems that can perceive and reason with human behavior, focusing on motion sequences, but it is incremental as it builds on existing neuro-symbolic approaches.
The paper tackles the problem of complex spatio-temporal reasoning over human motion sequences by proposing the HumanMotionQA task and dataset, and introduces NSPose, a neuro-symbolic method that outperforms all baselines on this task.
In order to build artificial intelligence systems that can perceive and reason with human behavior in the real world, we must first design models that conduct complex spatio-temporal reasoning over motion sequences. Moving towards this goal, we propose the HumanMotionQA task to evaluate complex, multi-step reasoning abilities of models on long-form human motion sequences. We generate a dataset of question-answer pairs that require detecting motor cues in small portions of motion sequences, reasoning temporally about when events occur, and querying specific motion attributes. In addition, we propose NSPose, a neuro-symbolic method for this task that uses symbolic reasoning and a modular design to ground motion through learning motion concepts, attribute neural operators, and temporal relations. We demonstrate the suitability of NSPose for the HumanMotionQA task, outperforming all baseline methods.