Modulating Pretrained Diffusion Models for Multimodal Image Synthesis
This provides a computationally efficient method for multimodal image synthesis, allowing user control over spatial layout, but it is incremental as it builds on existing pretrained models.
The paper tackles the problem of enabling conditional image synthesis with pretrained diffusion models without updating their parameters, achieving this by training a small module that modulates predictions during sampling using unseen 2D modalities like semantic maps, which requires only ~1% of the base model's parameters and limited training examples.
We present multimodal conditioning modules (MCM) for enabling conditional image synthesis using pretrained diffusion models. Previous multimodal synthesis works rely on training networks from scratch or fine-tuning pretrained networks, both of which are computationally expensive for large, state-of-the-art diffusion models. Our method uses pretrained networks but \textit{does not require any updates to the diffusion network's parameters}. MCM is a small module trained to modulate the diffusion network's predictions during sampling using 2D modalities (e.g., semantic segmentation maps, sketches) that were unseen during the original training of the diffusion model. We show that MCM enables user control over the spatial layout of the image and leads to increased control over the image generation process. Training MCM is cheap as it does not require gradients from the original diffusion net, consists of only $\sim$1$\%$ of the number of parameters of the base diffusion model, and is trained using only a limited number of training examples. We evaluate our method on unconditional and text-conditional models to demonstrate the improved control over the generated images and their alignment with respect to the conditioning inputs.