Watch-n-Patch: Unsupervised Learning of Actions and Relations
This work addresses the problem of understanding complex human activities for applications in assistive robotics, though it appears incremental as it builds on existing unsupervised learning approaches.
The paper tackles unsupervised modeling of composite human activities from videos by learning high-level co-occurrence and temporal relations between actions, resulting in a new probabilistic model that enables action segmentation, clustering, and a novel application for detecting forgotten actions called action patching.
There is a large variation in the activities that humans perform in their everyday lives. We consider modeling these composite human activities which comprises multiple basic level actions in a completely unsupervised setting. Our model learns high-level co-occurrence and temporal relations between the actions. We consider the video as a sequence of short-term action clips, which contains human-words and object-words. An activity is about a set of action-topics and object-topics indicating which actions are present and which objects are interacting with. We then propose a new probabilistic model relating the words and the topics. It allows us to model long-range action relations that commonly exist in the composite activities, which is challenging in previous works. We apply our model to the unsupervised action segmentation and clustering, and to a novel application that detects forgotten actions, which we call action patching. For evaluation, we contribute a new challenging RGB-D activity video dataset recorded by the new Kinect v2, which contains several human daily activities as compositions of multiple actions interacting with different objects. Moreover, we develop a robotic system that watches people and reminds people by applying our action patching algorithm. Our robotic setup can be easily deployed on any assistive robot.