Track and Caption Any Motion: Query-Free Motion Discovery and Description in Videos
This addresses the problem of video understanding for applications requiring motion analysis, such as surveillance or robotics, and is incremental as it builds on existing contrastive vision-language methods.
The paper tackles the problem of automatic video understanding in challenging conditions like occlusion and rapid movement by proposing TCAM, a motion-centric framework that discovers and describes motion patterns without user queries, achieving 58.4% video-to-text retrieval, 64.9 JF for spatial grounding, and 84.7% precision in discovering relevant expressions on the MeViS benchmark.
We propose Track and Caption Any Motion (TCAM), a motion-centric framework for automatic video understanding that discovers and describes motion patterns without user queries. Understanding videos in challenging conditions like occlusion, camouflage, or rapid movement often depends more on motion dynamics than static appearance. TCAM autonomously observes a video, identifies multiple motion activities, and spatially grounds each natural language description to its corresponding trajectory through a motion-field attention mechanism. Our key insight is that motion patterns, when aligned with contrastive vision-language representations, provide powerful semantic signals for recognizing and describing actions. Through unified training that combines global video-text alignment with fine-grained spatial correspondence, TCAM enables query-free discovery of multiple motion expressions via multi-head cross-attention. On the MeViS benchmark, TCAM achieves 58.4% video-to-text retrieval, 64.9 JF for spatial grounding, and discovers 4.8 relevant expressions per video with 84.7% precision, demonstrating strong cross-task generalization.