Deep Multi-Kernel Convolutional LSTM Networks and an Attention-Based Mechanism for Videos
This work addresses efficiency and accuracy challenges in video action recognition for computer vision applications, representing an incremental improvement over existing convolutional LSTM methods.
The paper tackled the trade-off between effectiveness and efficiency in convolutional LSTM networks for video action recognition by proposing a multi-kernel extension and an attention-based mechanism, resulting in improved accuracy on UCF-101 and Sports-1M datasets.
Action recognition greatly benefits motion understanding in video analysis. Recurrent networks such as long short-term memory (LSTM) networks are a popular choice for motion-aware sequence learning tasks. Recently, a convolutional extension of LSTM was proposed, in which input-to-hidden and hidden-to-hidden transitions are modeled through convolution with a single kernel. This implies an unavoidable trade-off between effectiveness and efficiency. Herein, we propose a new enhancement to convolutional LSTM networks that supports accommodation of multiple convolutional kernels and layers. This resembles a Network-in-LSTM approach, which improves upon the aforementioned concern. In addition, we propose an attention-based mechanism that is specifically designed for our multi-kernel extension. We evaluated our proposed extensions in a supervised classification setting on the UCF-101 and Sports-1M datasets, with the findings showing that our enhancements improve accuracy. We also undertook qualitative analysis to reveal the characteristics of our system and the convolutional LSTM baseline.