Diffusion Soup: Model Merging for Text-to-Image Diffusion Models
This provides a scalable solution for model merging in text-to-image generation, allowing efficient updates and style blending without retraining, though it is incremental in applying weight averaging to diffusion models.
The paper tackles the problem of merging text-to-image diffusion models trained on sharded data by averaging their weights, enabling training-free continual learning and unlearning with no extra costs. It shows that this method outperforms a model trained on all data combined, achieving improvements like a 30% increase in Image Reward on domain-sharded data and a 59% improvement on aesthetic data.
We present Diffusion Soup, a compartmentalization method for Text-to-Image Generation that averages the weights of diffusion models trained on sharded data. By construction, our approach enables training-free continual learning and unlearning with no additional memory or inference costs, since models corresponding to data shards can be added or removed by re-averaging. We show that Diffusion Soup samples from a point in weight space that approximates the geometric mean of the distributions of constituent datasets, which offers anti-memorization guarantees and enables zero-shot style mixing. Empirically, Diffusion Soup outperforms a paragon model trained on the union of all data shards and achieves a 30% improvement in Image Reward (.34 $\to$ .44) on domain sharded data, and a 59% improvement in IR (.37 $\to$ .59) on aesthetic data. In both cases, souping also prevails in TIFA score (respectively, 85.5 $\to$ 86.5 and 85.6 $\to$ 86.8). We demonstrate robust unlearning -- removing any individual domain shard only lowers performance by 1% in IR (.45 $\to$ .44) -- and validate our theoretical insights on anti-memorization using real data. Finally, we showcase Diffusion Soup's ability to blend the distinct styles of models finetuned on different shards, resulting in the zero-shot generation of hybrid styles.