From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models
This work suggests VDMs could serve as adaptable visual learners for future vision foundation models, though it builds incrementally on existing diffusion model capabilities.
The authors demonstrated that Video Diffusion Models (VDMs) can be repurposed for few-shot learning across diverse vision tasks by fine-tuning with minimal examples, achieving strong generalization without altering the core generative model.
Video Diffusion Models (VDMs) have emerged as powerful generative tools, capable of synthesizing high-quality spatiotemporal content. Yet, their potential goes far beyond mere video generation. We argue that the training dynamics of VDMs, driven by the need to model coherent sequences, naturally pushes them to internalize structured representations and an implicit understanding of the visual world. To probe the extent of this internal knowledge, we introduce a few-shot fine-tuning framework that repurposes VDMs for new tasks using only a handful of examples. Our method transforms each task into a visual transition, enabling the training of LoRA weights on short input-output sequences without altering the generative interface of a frozen VDM. Despite minimal supervision, the model exhibits strong generalization across diverse tasks, from low-level vision (for example, segmentation and pose estimation) to high-level reasoning (for example, on ARC-AGI). These results reframe VDMs as more than generative engines. They are adaptable visual learners with the potential to serve as the backbone for future foundation models in vision.