The Cow of Rembrandt - Analyzing Artistic Prompt Interpretation in Text-to-Image Models
This work addresses the problem of interpreting artistic concepts in generative models for researchers in AI and computer vision, but it is incremental as it builds on existing attention analysis methods.
The study investigated how text-to-image diffusion models internally represent content and style concepts in artistic prompts, finding that content tokens primarily influence object-related regions while style tokens affect background and texture areas, suggesting an emergent understanding of the distinction without explicit supervision.
Text-to-image diffusion models have demonstrated remarkable capabilities in generating artistic content by learning from billions of images, including popular artworks. However, the fundamental question of how these models internally represent concepts, such as content and style in paintings, remains unexplored. Traditional computer vision assumes content and style are orthogonal, but diffusion models receive no explicit guidance about this distinction during training. In this work, we investigate how transformer-based text-to-image diffusion models encode content and style concepts when generating artworks. We leverage cross-attention heatmaps to attribute pixels in generated images to specific prompt tokens, enabling us to isolate image regions influenced by content-describing versus style-describing tokens. Our findings reveal that diffusion models demonstrate varying degrees of content-style separation depending on the specific artistic prompt and style requested. In many cases, content tokens primarily influence object-related regions while style tokens affect background and texture areas, suggesting an emergent understanding of the content-style distinction. These insights contribute to our understanding of how large-scale generative models internally represent complex artistic concepts without explicit supervision. We share the code and dataset, together with an exploratory tool for visualizing attention maps at https://github.com/umilISLab/artistic-prompt-interpretation.