FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos
This addresses the problem of limited motion understanding in AI models for researchers and developers, offering a scalable solution for dataset creation, though it is incremental as it builds on existing methods like object detection and LLMs.
The paper tackles the scarcity of large-scale, fine-grained motion datasets by introducing FoundationMotion, an automated pipeline for curating such datasets, which when used to fine-tune models leads to substantial improvements, outperforming strong baselines like Gemini-2.5 Flash and Qwen2.5-VL-72B across diverse motion benchmarks.
Motion understanding is fundamental to physical reasoning, enabling models to infer dynamics and predict future states. However, state-of-the-art models still struggle on recent motion benchmarks, primarily due to the scarcity of large-scale, fine-grained motion datasets. Existing motion datasets are often constructed from costly manual annotation, severely limiting scalability. To address this challenge, we introduce FoundationMotion, a fully automated data curation pipeline that constructs large-scale motion datasets. Our approach first detects and tracks objects in videos to extract their trajectories, then leverages these trajectories and video frames with Large Language Models (LLMs) to generate fine-grained captions and diverse question-answer pairs about motion and spatial reasoning. Using datasets produced by this pipeline, we fine-tune open-source models including NVILA-Video-15B and Qwen2.5-7B, achieving substantial improvements in motion understanding without compromising performance on other tasks. Notably, our models outperform strong closed-source baselines like Gemini-2.5 Flash and large open-source models such as Qwen2.5-VL-72B across diverse motion understanding datasets and benchmarks. FoundationMotion thus provides a scalable solution for curating fine-grained motion datasets that enable effective fine-tuning of diverse models to enhance motion understanding and spatial reasoning capabilities.