Fashion Florence: Fine-Tuning Florence-2 for Structured Fashion Attribute Extraction
For fashion e-commerce and recommendation systems, this provides a lightweight, high-accuracy model for structured attribute extraction, though the task is domain-specific and the gains are incremental over existing vision-language models.
Fashion Florence fine-tunes Florence-2 with LoRA to extract structured fashion attributes (category, color, material, style, occasion) from clothing images, achieving 94.6% category accuracy and 63.0% material accuracy on a test set, outperforming GPT-4o-mini and Gemini 2.5 Flash while running at 0.77B parameters.
We present Fashion Florence, a Florence-2 vision-language model fine-tuned with LoRA to extract structured fashion attributes from clothing images. Given a single photograph, the model generates a JSON object containing category, color, material, style tags, and occasion tags, structured output suitable for direct programmatic consumption by downstream recommendation and retrieval systems. Fine-tuning data is derived from the iMaterialist Fashion dataset (228 labels), where we collapse fine-grained annotations into a compact 6-category, 16-color, 19-style schema via rule-based label engineering. We apply LoRA (r=16, alpha=32) to all decoder linear layers, training for 3 epochs on 3,688 examples. On a held-out test set of 461 images, Fashion Florence achieves 94.6% category accuracy and 63.0% material accuracy, compared to 89.3% / 43.3% for GPT-4o-mini and 87.4% for Gemini 2.5 Flash. Fashion Florence produces valid JSON in 99.8% of outputs while running at 0.77B parameters on a single GPU at zero marginal inference cost. Style tag F1 reaches 0.753 vs. 0.612 (Gemini) and 0.398 (GPT-4o-mini). The model is deployed as a Hugging Face Space and integrated into Loom, an open-source outfit recommendation system.