4DVLT: Dynamic Scene Understanding with Worldline-Centered Vision-Language Tracking
For researchers in vision-language tracking and 4D scene understanding, this work provides a new task and benchmark that integrates metric 3D motion with language grounding, showing significant gains over existing methods.
4DVLT introduces a worldline-centered task for 4D dynamic scene understanding from multi-view video, along with the Instruct-4D benchmark (129.4K QA pairs, 64.7K targets, 851 scenes). Their 4DTrack model achieves 62.68 TGA_Top1 on Instruct-4D, surpassing the best adapted VLT baseline by 19.62 points.
4D dynamic scene understanding requires grounding language to a persistent worldline that binds identity, metric 3D motion, and synchronized multi-view 2D projections. Existing paradigms capture only part of this structure: large multimodal models reason over rich visual evidence but rarely preserve metric topology, while vision-language tracking remains tied to fragmented 2D or 3D outputs and local continuation. We therefore introduce \textbf{4DVLT}, a worldline-centered task for instruction-conditioned 4D dynamic scene understanding in fully observed multi-view video, and \textbf{Instruct-4D}, a benchmark with 129.4K question-answer pairs, 64.7K target entities, 851 scenes, and 9 reasoning-oriented query types. To address this setting, we present \textbf{4DTrack}, which casts instruction-conditioned tracking as graph-conditioned worldline inference through an object-centric 4D state graph, metric-guided routing, bidirectional decoding, and kinematic calibration. On Instruct-4D, 4DTrack-Qwen3.5-9B reaches 62.68 $\mathrm{TGA}_{\mathrm{Top1}}$ and surpasses the best adapted VLT baseline by 19.62 points. These results show that worldline-centered modeling improves both target grounding and recovered worldline quality. The project page is available at https://github.com/mikubaka88/4DVLT.