Image Diffusion Models Exhibit Emergent Temporal Propagation in Videos
This provides a novel zero-shot object tracking method for video analysis, leveraging existing diffusion models without retraining.
The paper tackled the problem of object tracking in videos by reinterpreting self-attention maps from image diffusion models as semantic label propagation kernels, achieving state-of-the-art zero-shot performance on standard benchmarks.
Image diffusion models, though originally developed for image generation, implicitly capture rich semantic structures that enable various recognition and localization tasks beyond synthesis. In this work, we investigate their self-attention maps can be reinterpreted as semantic label propagation kernels, providing robust pixel-level correspondences between relevant image regions. Extending this mechanism across frames yields a temporal propagation kernel that enables zero-shot object tracking via segmentation in videos. We further demonstrate the effectiveness of test-time optimization strategies-DDIM inversion, textual inversion, and adaptive head weighting-in adapting diffusion features for robust and consistent label propagation. Building on these findings, we introduce DRIFT, a framework for object tracking in videos leveraging a pretrained image diffusion model with SAM-guided mask refinement, achieving state-of-the-art zero-shot performance on standard video object segmentation benchmarks.