Facial Dynamics in Video: Instruction Tuning for Improved Facial Expression Perception and Contextual Awareness
It addresses the problem of limited datasets and visual token capacity for video MLLMs in facial expression perception, offering incremental improvements for applications in video understanding and human-computer interaction.
This paper tackles the challenges of facial expression captioning in videos by introducing a new instruction-following dataset with 5,033 video clips and over 700,000 tokens, and proposing FaceTrack-MM, which achieves superior performance in tracking faces and focusing on expressions in multi-person scenarios.
Facial expression captioning has found widespread application across various domains. Recently, the emergence of video Multimodal Large Language Models (MLLMs) has shown promise in general video understanding tasks. However, describing facial expressions within videos poses two major challenges for these models: (1) the lack of adequate datasets and benchmarks, and (2) the limited visual token capacity of video MLLMs. To address these issues, this paper introduces a new instruction-following dataset tailored for dynamic facial expression caption. The dataset comprises 5,033 high-quality video clips annotated manually, containing over 700,000 tokens. Its purpose is to improve the capability of video MLLMs to discern subtle facial nuances. Furthermore, we propose FaceTrack-MM, which leverages a limited number of tokens to encode the main character's face. This model demonstrates superior performance in tracking faces and focusing on the facial expressions of the main characters, even in intricate multi-person scenarios. Additionally, we introduce a novel evaluation metric combining event extraction, relation classification, and the longest common subsequence (LCS) algorithm to assess the content consistency and temporal sequence consistency of generated text. Moreover, we present FEC-Bench, a benchmark designed to assess the performance of existing video MLLMs in this specific task. All data and source code will be made publicly available.