Text-driven Visual Synthesis with Latent Diffusion Prior
This work addresses the need for more effective use of diffusion models in visual synthesis, though it appears incremental by building on existing priors.
The paper tackles the problem of leveraging latent diffusion models as image priors for visual synthesis tasks, achieving improved performance in text-to-3D, StyleGAN adaptation, and layered image editing compared to baselines.
There has been tremendous progress in large-scale text-to-image synthesis driven by diffusion models enabling versatile downstream applications such as 3D object synthesis from texts, image editing, and customized generation. We present a generic approach using latent diffusion models as powerful image priors for various visual synthesis tasks. Existing methods that utilize such priors fail to use these models' full capabilities. To improve this, our core ideas are 1) a feature matching loss between features from different layers of the decoder to provide detailed guidance and 2) a KL divergence loss to regularize the predicted latent features and stabilize the training. We demonstrate the efficacy of our approach on three different applications, text-to-3D, StyleGAN adaptation, and layered image editing. Extensive results show our method compares favorably against baselines.