TryOnDiffusion: A Tale of Two UNets
This addresses the challenge of virtual try-on for e-commerce and fashion applications, enabling realistic garment visualization across different body types and poses, though it is an incremental improvement over existing methods.
The paper tackles the problem of generating photorealistic visualizations of garments on a person with significant pose and shape changes, achieving state-of-the-art performance by preserving garment details and warping them effectively in a single network.
Given two images depicting a person and a garment worn by another person, our goal is to generate a visualization of how the garment might look on the input person. A key challenge is to synthesize a photorealistic detail-preserving visualization of the garment, while warping the garment to accommodate a significant body pose and shape change across the subjects. Previous methods either focus on garment detail preservation without effective pose and shape variation, or allow try-on with the desired shape and pose but lack garment details. In this paper, we propose a diffusion-based architecture that unifies two UNets (referred to as Parallel-UNet), which allows us to preserve garment details and warp the garment for significant pose and body change in a single network. The key ideas behind Parallel-UNet include: 1) garment is warped implicitly via a cross attention mechanism, 2) garment warp and person blend happen as part of a unified process as opposed to a sequence of two separate tasks. Experimental results indicate that TryOnDiffusion achieves state-of-the-art performance both qualitatively and quantitatively.