Forecasting Future Action Sequences with Neural Memory Networks
This work addresses a challenging task in video understanding for applications like robotics or surveillance, but it appears incremental as it builds on existing neural memory networks for a specific domain.
The paper tackles the problem of forecasting future action sequences by proposing a neural memory network framework that captures short-term and long-term relationships, achieving state-of-the-art performance with significant improvements on the Breakfast and 50 Salads datasets.
We propose a novel neural memory network based framework for future action sequence forecasting. This is a challenging task where we have to consider short-term, within sequence relationships as well as relationships in between sequences, to understand how sequences of actions evolve over time. To capture these relationships effectively, we introduce neural memory networks to our modelling scheme. We show the significance of using two input streams, the observed frames and the corresponding action labels, which provide different information cues for our prediction task. Furthermore, through the proposed method we effectively map the long-term relationships among individual input sequences through separate memory modules, which enables better fusion of the salient features. Our method outperforms the state-of-the-art approaches by a large margin on two publicly available datasets: Breakfast and 50 Salads.