Generative Spatiotemporal Data Augmentation
This work addresses data augmentation challenges for spatiotemporal applications, particularly in domains with scarce annotations, though it is incremental as it builds on existing generative methods.
The paper tackles the problem of data scarcity in spatiotemporal tasks by using video foundation models to generate realistic 3D spatial and temporal variations from images, resulting in consistent performance gains in low-data settings like UAV-captured imagery.
We explore spatiotemporal data augmentation using video foundation models to diversify both camera viewpoints and scene dynamics. Unlike existing approaches based on simple geometric transforms or appearance perturbations, our method leverages off-the-shelf video diffusion models to generate realistic 3D spatial and temporal variations from a given image dataset. Incorporating these synthesized video clips as supplemental training data yields consistent performance gains in low-data settings, such as UAV-captured imagery where annotations are scarce. Beyond empirical improvements, we provide practical guidelines for (i) choosing an appropriate spatiotemporal generative setup, (ii) transferring annotations to synthetic frames, and (iii) addressing disocclusion - regions newly revealed and unlabeled in generated views. Experiments on COCO subsets and UAV-captured datasets show that, when applied judiciously, spatiotemporal augmentation broadens the data distribution along axes underrepresented by traditional and prior generative methods, offering an effective lever for improving model performance in data-scarce regimes.