Diffusion Cocktail: Mixing Domain-Specific Diffusion Models for Diversified Image Generations
This work addresses the need for more flexible and creative image generation by leveraging existing fine-tuned models, offering incremental improvements in style transfer and manipulation for users in AI art and content creation.
The authors tackled the problem of generating novel images by combining multiple domain-specific diffusion models, proposing Diffusion Cocktail (Ditail) as a training-free method that transfers style and content information to achieve diversified generations with fine-grained control.
Diffusion models, capable of high-quality image generation, receive unparalleled popularity for their ease of extension. Active users have created a massive collection of domain-specific diffusion models by fine-tuning base models on self-collected datasets. Recent work has focused on improving a single diffusion model by uncovering semantic and visual information encoded in various architecture components. However, those methods overlook the vastly available set of fine-tuned diffusion models and, therefore, miss the opportunity to utilize their combined capacity for novel generation. In this work, we propose Diffusion Cocktail (Ditail), a training-free method that transfers style and content information between multiple diffusion models. This allows us to perform diversified generations using a set of diffusion models, resulting in novel images unobtainable by a single model. Ditail also offers fine-grained control of the generation process, which enables flexible manipulations of styles and contents. With these properties, Ditail excels in numerous applications, including style transfer guided by diffusion models, novel-style image generation, and image manipulation via prompts or collage inputs.