Style-Friendly SNR Sampler for Style-Driven Generation
This addresses the limitation in creating unique style templates for users in style-driven generation, though it is incremental as it builds on existing fine-tuning methods.
The paper tackles the problem of text-to-image diffusion models struggling to learn new personalized styles from reference images, proposing a Style-friendly SNR sampler that shifts noise distribution during fine-tuning to improve style alignment, resulting in enhanced generation of novel styles for personalized content creation.
Recent text-to-image diffusion models generate high-quality images but struggle to learn new, personalized styles, which limits the creation of unique style templates. In style-driven generation, users typically supply reference images exemplifying the desired style, together with text prompts that specify desired stylistic attributes. Previous approaches popularly rely on fine-tuning, yet it often blindly utilizes objectives and noise level distributions from pre-training without adaptation. We discover that stylistic features predominantly emerge at higher noise levels, leading current fine-tuning methods to exhibit suboptimal style alignment. We propose the Style-friendly SNR sampler, which aggressively shifts the signal-to-noise ratio (SNR) distribution toward higher noise levels during fine-tuning to focus on noise levels where stylistic features emerge. This enhances models' ability to capture novel styles indicated by reference images and text prompts. We demonstrate improved generation of novel styles that cannot be adequately described solely with a text prompt, enabling the creation of new style templates for personalized content creation.