Gen4U: Unifying Video Generation and Understanding via Diffusion
For researchers in video understanding and generation, this work demonstrates that frozen video diffusion models can serve as universal video encoders, unifying two previously separate paradigms.
The paper shows that large-scale video diffusion models have highly structured latent spaces that encode both low-level geometry and high-level semantics. By repurposing these generative representations without fine-tuning, Gen4U achieves strong performance across diverse video understanding tasks (classification, depth estimation, pose estimation, captioning) while preserving generation quality.
Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics. We demonstrate that state-of-the-art video diffusion models overcome this limitation. By systematically probing their intermediate activations using recent mutual-kNN alignment metrics, we reveal a highly structured latent space where visual representations evolve across both network depth and noise levels. We show that while moderate noise levels yield linearly separable global semantics, fine-grained details persist at lower noise levels but become spatially scattered, requiring attention mechanisms to decode. Building on these insights, we introduce Gen4U (Generation for Understanding), a framework that repurposes these generative representations with a single forward pass. Our experiments establish that frozen, large-scale video diffusion models function as highly competitive video encoders across a wide spectrum of tasks, spanning semantic and non-semantic objectives (video classification, depth estimation, camera pose estimation, image and video captioning). Bypassing fine-tuning, Gen4U unifies the generation and understanding paradigms, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.