SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
This addresses the data scarcity problem for researchers developing spatial reasoning capabilities in video language models, though it's an incremental improvement over existing spatial training approaches.
The researchers tackled the bottleneck of obtaining diverse real-world video data with precise spatial annotations for training multimodal language models by developing SIMS-V, a framework that generates spatially-rich video training data using 3D simulators. Their 7B-parameter model fine-tuned on just 25K simulated examples outperformed a 72B baseline and achieved competitive performance with proprietary models on real-world spatial reasoning benchmarks.
Despite impressive high-level video comprehension, multimodal language models struggle with spatial reasoning across time and space. While current spatial training approaches rely on real-world video data, obtaining diverse footage with precise spatial annotations remains a bottleneck. To alleviate this bottleneck, we present SIMS-V -- a systematic data-generation framework that leverages the privileged information of 3D simulators to create spatially-rich video training data for multimodal language models. Using this framework, we investigate which properties of simulated data drive effective real-world transfer through systematic ablations of question types, mixes, and scales. We identify a minimal set of three question categories (metric measurement, perspective-dependent reasoning, and temporal tracking) that prove most effective for developing transferable spatial intelligence, outperforming comprehensive coverage despite using fewer question types. These insights enable highly efficient training: our 7B-parameter video LLM fine-tuned on just 25K simulated examples outperforms the larger 72B baseline and achieves competitive performance with proprietary models on rigorous real-world spatial reasoning benchmarks. Our approach demonstrates robust generalization, maintaining performance on general video understanding while showing substantial improvements on embodied and real-world spatial tasks.