GTASA: Ground Truth Annotations for Spatiotemporal Analysis, Evaluation and Training of Video Models
For researchers in video generation and spatiotemporal reasoning, GTASA offers a benchmark with exact ground truth to evaluate and train models, addressing the lack of physically plausible and semantically faithful video data.
GTASA provides a corpus of multi-actor videos with ground truth spatial and temporal annotations, enabling evaluation of video models. The system outperforms neural generators in physical plausibility and semantic alignment, and reveals that self-supervised encoders encode spatial structure better than VLM visual encoders.
Generating complex multi-actor scenario videos remains difficult even for state-of-the-art neural generators, while evaluating them is hard due to the lack of ground truth for physical plausibility and semantic faithfulness. We introduce GTASA, a corpus of multi-actor videos with per-frame spatial relation graphs and event-level temporal mappings, and the system that produced it based on Graphs of Events in Space and Time (GEST): GEST-Engine. We compare our method with both open and closed source neural generators and prove both qualitatively (human evaluation of physical validity and semantic alignment) and quantitatively (via training video captioning models) the clear advantages of our method. Probing four frozen video encoders across 11 spatiotemporal reasoning tasks enabled by GTASA's exact 3D ground truth reveals that self-supervised encoders encode spatial structure significantly better than VLM visual encoders.