Visual Style Prompting with Swapping Self-Attention
This addresses the challenge of style consistency in diffusion models for content creators, offering a novel, fine-tuning-free solution that is incremental in improving style transfer techniques.
The paper tackles the problem of achieving controlled text-to-image generation with consistent visual styles without fine-tuning, proposing a method that swaps key and value in self-attention layers to maintain style fidelity, resulting in superior performance in style reflection and text matching compared to existing approaches.
In the evolving domain of text-to-image generation, diffusion models have emerged as powerful tools in content creation. Despite their remarkable capability, existing models still face challenges in achieving controlled generation with a consistent style, requiring costly fine-tuning or often inadequately transferring the visual elements due to content leakage. To address these challenges, we propose a novel approach, \ours, to produce a diverse range of images while maintaining specific style elements and nuances. During the denoising process, we keep the query from original features while swapping the key and value with those from reference features in the late self-attention layers. This approach allows for the visual style prompting without any fine-tuning, ensuring that generated images maintain a faithful style. Through extensive evaluation across various styles and text prompts, our method demonstrates superiority over existing approaches, best reflecting the style of the references and ensuring that resulting images match the text prompts most accurately. Our project page is available https://curryjung.github.io/VisualStylePrompt/.