A Comprehensive Study of Deep Video Action Recognition
This survey paper helps researchers and practitioners in video understanding by organizing existing knowledge, identifying challenges, and suggesting future research directions in video action recognition.
This paper provides a comprehensive survey of over 200 deep learning papers on video action recognition, covering 17 datasets, chronological model advancements from early adaptations to compute-efficient designs, and benchmarking popular methods on representative datasets.
Video action recognition is one of the representative tasks for video understanding. Over the last decade, we have witnessed great advancements in video action recognition thanks to the emergence of deep learning. But we also encountered new challenges, including modeling long-range temporal information in videos, high computation costs, and incomparable results due to datasets and evaluation protocol variances. In this paper, we provide a comprehensive survey of over 200 existing papers on deep learning for video action recognition. We first introduce the 17 video action recognition datasets that influenced the design of models. Then we present video action recognition models in chronological order: starting with early attempts at adapting deep learning, then to the two-stream networks, followed by the adoption of 3D convolutional kernels, and finally to the recent compute-efficient models. In addition, we benchmark popular methods on several representative datasets and release code for reproducibility. In the end, we discuss open problems and shed light on opportunities for video action recognition to facilitate new research ideas.