From Image to Video: An Empirical Study of Diffusion Representations
This work addresses the gap in understanding video diffusion representations for visual tasks, offering insights into temporal information's role, though it is incremental as it builds on existing diffusion model research.
The study compared video and image diffusion models on visual understanding tasks, finding that video models consistently outperformed image models, with varying degrees of superiority across tasks.
Diffusion models have revolutionized generative modeling, enabling unprecedented realism in image and video synthesis. This success has sparked interest in leveraging their representations for visual understanding tasks. While recent works have explored this potential for image generation, the visual understanding capabilities of video diffusion models remain largely uncharted. To address this gap, we systematically compare the same model architecture trained for video versus image generation, analyzing the performance of their latent representations on various downstream tasks including image classification, action recognition, depth estimation, and tracking. Results show that video diffusion models consistently outperform their image counterparts, though we find a striking range in the extent of this superiority. We further analyze features extracted from different layers and with varying noise levels, as well as the effect of model size and training budget on representation and generation quality. This work marks the first direct comparison of video and image diffusion objectives for visual understanding, offering insights into the role of temporal information in representation learning.