ViLP: Knowledge Exploration using Vision, Language, and Pose Embeddings for Video Action Recognition
It addresses the challenge of recognizing a large number of human actions in videos for computer vision applications, presenting a novel multi-modal approach.
The paper tackles video action recognition by introducing a unified framework that combines pose, visual, and text modalities, achieving accuracies of 92.81% on UCF-101 and 73.02% on HMDB-51 without pre-training, and 96.11% and 75.75% with pre-training.
Video Action Recognition (VAR) is a challenging task due to its inherent complexities. Though different approaches have been explored in the literature, designing a unified framework to recognize a large number of human actions is still a challenging problem. Recently, Multi-Modal Learning (MML) has demonstrated promising results in this domain. In literature, 2D skeleton or pose modality has often been used for this task, either independently or in conjunction with the visual information (RGB modality) present in videos. However, the combination of pose, visual information, and text attributes has not been explored yet, though text and pose attributes independently have been proven to be effective in numerous computer vision tasks. In this paper, we present the first pose augmented Vision-language model (VLM) for VAR. Notably, our scheme achieves an accuracy of 92.81% and 73.02% on two popular human video action recognition benchmark datasets, UCF-101 and HMDB-51, respectively, even without any video data pre-training, and an accuracy of 96.11% and 75.75% after kinetics pre-training.