WildVidFit: Video Virtual Try-On in the Wild via Image-Based Controlled Diffusion Models
This addresses the problem of realistic video try-on for applications like e-commerce and social media, offering a more efficient alternative to traditional methods, though it is incremental in improving temporal coherence within an image-based framework.
The paper tackles video virtual try-on by generating realistic sequences that preserve garment identity and adapt to human motion, using an image-based controlled diffusion model trained on still images, which achieves fluid and coherent video generation as demonstrated on datasets like VITON-HD and DressCode.
Video virtual try-on aims to generate realistic sequences that maintain garment identity and adapt to a person's pose and body shape in source videos. Traditional image-based methods, relying on warping and blending, struggle with complex human movements and occlusions, limiting their effectiveness in video try-on applications. Moreover, video-based models require extensive, high-quality data and substantial computational resources. To tackle these issues, we reconceptualize video try-on as a process of generating videos conditioned on garment descriptions and human motion. Our solution, WildVidFit, employs image-based controlled diffusion models for a streamlined, one-stage approach. This model, conditioned on specific garments and individuals, is trained on still images rather than videos. It leverages diffusion guidance from pre-trained models including a video masked autoencoder for segment smoothness improvement and a self-supervised model for feature alignment of adjacent frame in the latent space. This integration markedly boosts the model's ability to maintain temporal coherence, enabling more effective video try-on within an image-based framework. Our experiments on the VITON-HD and DressCode datasets, along with tests on the VVT and TikTok datasets, demonstrate WildVidFit's capability to generate fluid and coherent videos. The project page website is at wildvidfit-project.github.io.