Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency
This work addresses the problem of applying latent diffusion models to inverse problems for researchers and practitioners in fields like medical imaging, offering a more efficient alternative to pixel-space models, though it is incremental as it builds on existing latent diffusion frameworks.
The authors tackled the challenge of using latent diffusion models for solving inverse problems by proposing ReSample, an algorithm that incorporates hard data consistency and a novel resampling scheme, achieving superior performance over state-of-the-art methods in linear and nonlinear inverse problems for natural and medical images.
Diffusion models have recently emerged as powerful generative priors for solving inverse problems. However, training diffusion models in the pixel space are both data-intensive and computationally demanding, which restricts their applicability as priors for high-dimensional real-world data such as medical images. Latent diffusion models, which operate in a much lower-dimensional space, offer a solution to these challenges. However, incorporating latent diffusion models to solve inverse problems remains a challenging problem due to the nonlinearity of the encoder and decoder. To address these issues, we propose \textit{ReSample}, an algorithm that can solve general inverse problems with pre-trained latent diffusion models. Our algorithm incorporates data consistency by solving an optimization problem during the reverse sampling process, a concept that we term as hard data consistency. Upon solving this optimization problem, we propose a novel resampling scheme to map the measurement-consistent sample back onto the noisy data manifold and theoretically demonstrate its benefits. Lastly, we apply our algorithm to solve a wide range of linear and nonlinear inverse problems in both natural and medical images, demonstrating that our approach outperforms existing state-of-the-art approaches, including those based on pixel-space diffusion models.