Prediction and Description of Near-Future Activities in Video
This work addresses the challenging problem of anticipating future activities in video, which has important applications requiring anticipatory response, but it is incremental as it builds on existing recognition and captioning methods.
The paper tackles the problem of predicting and describing near-future activities in video, where no frames of the predicted activities are observed, by proposing a system that infers labels and captions for sequences of future activities. The results show that the label prediction approach achieves comparable performance with state-of-the-art methods, and the captioning framework outperforms state-of-the-art methods on four activity analysis datasets and a video description dataset.
Most of the existing works on human activity analysis focus on recognition or early recognition of the activity labels from complete or partial observations. Similarly, almost all of the existing video captioning approaches focus on the observed events in videos. Predicting the labels and the captions of future activities where no frames of the predicted activities have been observed is a challenging problem, with important applications that require anticipatory response. In this work, we propose a system that can infer the labels and the captions of a sequence of future activities. Our proposed network for label prediction of a future activity sequence has three branches where the first branch takes visual features from the objects present in the scene, the second branch takes observed sequential activity features, and the third branch captures the last observed activity features. The predicted labels and the observed scene context are then mapped to meaningful captions using a sequence-to-sequence learning-based method. Experiments on four challenging activity analysis datasets and a video description dataset demonstrate that our label prediction approach achieves comparable performance with the state-of-the-arts and our captioning framework outperform the state-of-the-arts.