Learning Representations for Predicting Future Activities
This work addresses future activity prediction for applications like robotics or video analysis, but it appears incremental as it builds on existing methods for representation learning and multi-hypotheses prediction.
The paper tackles the problem of predicting future activities by learning environment dynamics embeddings in a self-supervised way, using a multi-hypotheses scheme to handle ambiguities, and demonstrates results by classifying activities on Epic-Kitchens and Breakfast datasets and generating captions.
Foreseeing the future is one of the key factors of intelligence. It involves understanding of the past and current environment as well as decent experience of its possible dynamics. In this work, we address future prediction at the abstract level of activities. We propose a network module for learning embeddings of the environment's dynamics in a self-supervised way. To take the ambiguities and high variances in the future activities into account, we use a multi-hypotheses scheme that can represent multiple futures. We demonstrate the approach by classifying future activities on the Epic-Kitchens and Breakfast datasets. Moreover, we generate captions that describe the future activities