SIFT-VTON: Geometric Correspondence Supervision on Cross-Attention for Virtual Try-On
For virtual try-on applications, this work addresses the problem of preserving fine garment details by integrating classical geometric correspondence into diffusion models, offering a practical improvement over purely implicit learning approaches.
SIFT-VTON improves diffusion-based virtual try-on by using SIFT keypoint matching to supervise cross-attention layers, achieving better preservation of fine details like text and patterns. On VITON-HD, it shows significant improvements in unpaired metrics while maintaining competitive paired reconstruction.
Diffusion-based virtual try-on methods achieve photorealistic synthesis through cross-attention mechanisms that transfer garment features to target body regions. However, these approaches rely on implicit learning of spatial correspondences, struggling to preserve fine details such as text and illustrations. We propose a novel approach, which we call SIFT-VTON, that utilizes SIFT keypoint matching to provide explicit geometric guidance for diffusion-based virtual try-on. Our method applies domain-specific filtering to SIFT keypoint matches between garment and person images, then converts these correspondences into spatial probability distributions that supervise cross-attention layers during training. This explicit supervision guides the model to learn precise spatial alignment, concentrating attention on geometrically consistent garment regions. Experiments on the VITON-HD dataset demonstrate significant improvements on unpaired metrics while maintaining competitive paired reconstruction metrics. Qualitative comparisons show superior preservation of text clarity and pattern alignment. Attention visualizations confirm that our method produces sharply focused attention on relevant garment details. This work demonstrates that classical geometric correspondence methods can effectively enhance modern diffusion models for conditional synthesis tasks. The source code will be available at https://github.com/takesukeDS/SIFT-VTON.