Diving Deep into the Motion Representation of Video-Text Models
This work addresses the need for better motion understanding in video-text models, which is incremental as it builds on existing retrieval tasks with new descriptions.
The paper tackled the problem of whether video-text models understand motion in videos by evaluating them on motion description retrieval tasks, finding they perform far behind human experts on two action datasets, and introduced a method using GPT-4 generated motion descriptions to improve performance, which proved effective.
Videos are more informative than images because they capture the dynamics of the scene. By representing motion in videos, we can capture dynamic activities. In this work, we introduce GPT-4 generated motion descriptions that capture fine-grained motion descriptions of activities and apply them to three action datasets. We evaluated several video-text models on the task of retrieval of motion descriptions. We found that they fall far behind human expert performance on two action datasets, raising the question of whether video-text models understand motion in videos. To address it, we introduce a method of improving motion understanding in video-text models by utilizing motion descriptions. This method proves to be effective on two action datasets for the motion description retrieval task. The results draw attention to the need for quality captions involving fine-grained motion information in existing datasets and demonstrate the effectiveness of the proposed pipeline in understanding fine-grained motion during video-text retrieval.