ActionArt: Advancing Multimodal Large Models for Fine-Grained Human-Centric Video Understanding
This work addresses the need for better human-centric AI applications by providing a dataset and evaluation framework, but it is incremental as it builds on existing multimodal models with proxy tasks.
The paper tackles the problem of fine-grained human action and pose understanding in videos by introducing ActionArt, a dataset with detailed annotations, and finds that current large multimodal models fall short due to data scarcity, with proxy tasks reducing the performance gap by leveraging automatically generated data.
Fine-grained understanding of human actions and poses in videos is essential for human-centric AI applications. In this work, we introduce ActionArt, a fine-grained video-caption dataset designed to advance research in human-centric multimodal understanding. Our dataset comprises thousands of videos capturing a broad spectrum of human actions, human-object interactions, and diverse scenarios, each accompanied by detailed annotations that meticulously label every limb movement. We develop eight sub-tasks to evaluate the fine-grained understanding capabilities of existing large multimodal models across different dimensions. Experimental results indicate that, while current large multimodal models perform commendably on various tasks, they often fall short in achieving fine-grained understanding. We attribute this limitation to the scarcity of meticulously annotated data, which is both costly and difficult to scale manually. Since manual annotations are costly and hard to scale, we propose proxy tasks to enhance the model perception ability in both spatial and temporal dimensions. These proxy tasks are carefully crafted to be driven by data automatically generated from existing MLLMs, thereby reducing the reliance on costly manual labels. Experimental results show that the proposed proxy tasks significantly narrow the gap toward the performance achieved with manually annotated fine-grained data.