FashionSD-X: Multimodal Fashion Garment Synthesis using Latent Diffusion
This work addresses the need for more interactive and personalized fashion design tools for designers and the fashion industry, though it is incremental as it builds on existing diffusion models and datasets.
The paper tackled the problem of generating fashion garment images from multimodal inputs like text and sketches, using a latent diffusion model with ControlNet and LoRA fine-tuning, and achieved significant improvements over traditional stable diffusion models as measured by metrics such as FID, CLIP Score, and KID.
The rapid evolution of the fashion industry increasingly intersects with technological advancements, particularly through the integration of generative AI. This study introduces a novel generative pipeline designed to transform the fashion design process by employing latent diffusion models. Utilizing ControlNet and LoRA fine-tuning, our approach generates high-quality images from multimodal inputs such as text and sketches. We leverage and enhance state-of-the-art virtual try-on datasets, including Multimodal Dress Code and VITON-HD, by integrating sketch data. Our evaluation, utilizing metrics like FID, CLIP Score, and KID, demonstrates that our model significantly outperforms traditional stable diffusion models. The results not only highlight the effectiveness of our model in generating fashion-appropriate outputs but also underscore the potential of diffusion models in revolutionizing fashion design workflows. This research paves the way for more interactive, personalized, and technologically enriched methodologies in fashion design and representation, bridging the gap between creative vision and practical application.