LatentCRF: Continuous CRF for Efficient Latent Diffusion
This addresses the speed bottleneck for users of LDMs in image generation, though it is incremental as it builds on existing LDM frameworks.
The paper tackles the high latency of Latent Diffusion Models (LDMs) by introducing LatentCRF, a continuous Conditional Random Field layer that models latent vector relationships, achieving a 33% increase in inference efficiency without quality or diversity loss.
Latent Diffusion Models (LDMs) produce high-quality, photo-realistic images, however, the latency incurred by multiple costly inference iterations can restrict their applicability. We introduce LatentCRF, a continuous Conditional Random Field (CRF) model, implemented as a neural network layer, that models the spatial and semantic relationships among the latent vectors in the LDM. By replacing some of the computationally-intensive LDM inference iterations with our lightweight LatentCRF, we achieve a superior balance between quality, speed and diversity. We increase inference efficiency by 33% with no loss in image quality or diversity compared to the full LDM. LatentCRF is an easy add-on, which does not require modifying the LDM.