Studying Image Diffusion Features for Zero-Shot Video Object Segmentation
This addresses video object segmentation for researchers and practitioners by enabling zero-shot performance without costly training, though it is incremental in applying diffusion models to this task.
The paper tackled Zero-Shot Video Object Segmentation by extracting features from diffusion models without fine-tuning, achieving state-of-the-art results on DAVIS-17 and MOSE datasets and matching models trained on expensive segmentation data.
This paper investigates the use of large-scale diffusion models for Zero-Shot Video Object Segmentation (ZS-VOS) without fine-tuning on video data or training on any image segmentation data. While diffusion models have demonstrated strong visual representations across various tasks, their direct application to ZS-VOS remains underexplored. Our goal is to find the optimal feature extraction process for ZS-VOS by identifying the most suitable time step and layer from which to extract features. We further analyze the affinity of these features and observe a strong correlation with point correspondences. Through extensive experiments on DAVIS-17 and MOSE, we find that diffusion models trained on ImageNet outperform those trained on larger, more diverse datasets for ZS-VOS. Additionally, we highlight the importance of point correspondences in achieving high segmentation accuracy, and we yield state-of-the-art results in ZS-VOS. Finally, our approach performs on par with models trained on expensive image segmentation datasets.