A LoRA is Worth a Thousand Pictures
This addresses the challenge of retrieving and attributing customized models in real-world scenarios where training data is unknown, though it is incremental as it builds on existing LoRA and PEFT methods.
The paper tackled the problem of describing artistic styles in text-to-image generation by showing that LoRA weights alone can effectively represent styles without needing original training images or additional generation, achieving better clustering performance than traditional features like CLIP and DINO with strong quantitative similarities.
Recent advances in diffusion models and parameter-efficient fine-tuning (PEFT) have made text-to-image generation and customization widely accessible, with Low Rank Adaptation (LoRA) able to replicate an artist's style or subject using minimal data and computation. In this paper, we examine the relationship between LoRA weights and artistic styles, demonstrating that LoRA weights alone can serve as an effective descriptor of style, without the need for additional image generation or knowledge of the original training set. Our findings show that LoRA weights yield better performance in clustering of artistic styles compared to traditional pre-trained features, such as CLIP and DINO, with strong structural similarities between LoRA-based and conventional image-based embeddings observed both qualitatively and quantitatively. We identify various retrieval scenarios for the growing collection of customized models and show that our approach enables more accurate retrieval in real-world settings where knowledge of the training images is unavailable and additional generation is required. We conclude with a discussion on potential future applications, such as zero-shot LoRA fine-tuning and model attribution.