Generalist Forecasting with Frozen Video Models via Latent Diffusion
This work addresses forecasting for general-purpose systems, but it is incremental as it builds on existing vision models and generative techniques.
The paper tackles the problem of generalist forecasting across multiple levels of abstraction by identifying a correlation between vision models' perceptual ability and forecasting performance, and introduces a framework using latent diffusion models on frozen backbones, achieving results evaluated on nine models and four tasks.
Forecasting what will happen next is a critical skill for general-purpose systems that plan or act in the world at different levels of abstraction. In this paper, we identify a strong correlation between a vision model's perceptual ability and its generalist forecasting performance over short time horizons. This trend holds across a diverse set of pretrained models-including those trained generatively-and across multiple levels of abstraction, from raw pixels to depth, point tracks, and object motion. The result is made possible by a novel generalist forecasting framework that operates on any frozen vision backbone: we train latent diffusion models to forecast future features in the frozen representation space, which are then decoded via lightweight, task-specific readouts. To enable consistent evaluation across tasks, we introduce distributional metrics that compare distributional properties directly in the space of downstream tasks and apply this framework to nine models and four tasks. Our results highlight the value of bridging representation learning and generative modeling for temporally grounded video understanding.