A Probabilistic Semi-Supervised Approach to Multi-Task Human Activity Modeling
This addresses the challenge of multi-task human activity modeling for applications in video analysis, though it appears incremental as it builds on existing probabilistic and latent variable methods.
The paper tackles the problem of modeling human activity from video data by developing a semi-supervised probabilistic deep latent variable model that handles multiple tasks like action classification, detection, prediction, and motion synthesis simultaneously, showing it outperforms or matches task-specific state-of-the-art models.
Human behavior is a continuous stochastic spatio-temporal process which is governed by semantic actions and affordances as well as latent factors. Therefore, video-based human activity modeling is concerned with a number of tasks such as inferring current and future semantic labels, predicting future continuous observations as well as imagining possible future label and feature sequences. In this paper we present a semi-supervised probabilistic deep latent variable model that can represent both discrete labels and continuous observations as well as latent dynamics over time. This allows the model to solve several tasks at once without explicit fine-tuning. We focus here on the tasks of action classification, detection, prediction and anticipation as well as motion prediction and synthesis based on 3D human activity data recorded with Kinect. We further extend the model to capture hierarchical label structure and to model the dependencies between multiple entities, such as a human and objects. Our experiments demonstrate that our principled approach to human activity modeling can be used to detect current and anticipate future semantic labels and to predict and synthesize future label and feature sequences. When comparing our model to state-of-the-art approaches, which are specifically designed for e.g. action classification, we find that our probabilistic formulation outperforms or is comparable to these task specific models.