Efficient Zero-Shot Inpainting with Decoupled Diffusion Guidance
For practitioners of image inpainting, this work offers a more efficient zero-shot alternative to existing diffusion-based methods, reducing computational burden without sacrificing quality.
The paper tackles the high memory and runtime overhead of zero-shot diffusion-based inpainting by proposing a new likelihood surrogate that avoids backpropagation through the denoiser. The method achieves strong observation consistency and high-quality reconstructions while significantly reducing inference cost.
Diffusion models have emerged as powerful priors for image editing tasks such as inpainting and local modification, where the objective is to generate realistic content that remains consistent with observed regions. In particular, zero-shot approaches that leverage a pretrained diffusion model, without any retraining, have been shown to achieve highly effective reconstructions. However, state-of-the-art zero-shot methods typically rely on a sequence of surrogate likelihood functions, whose scores are used as proxies for the ideal score. This procedure however requires vector-Jacobian products through the denoiser at every reverse step, introducing significant memory and runtime overhead. To address this issue, we propose a new likelihood surrogate that yields simple and efficient to sample Gaussian posterior transitions, sidestepping the backpropagation through the denoiser network. Our extensive experiments show that our method achieves strong observation consistency compared with fine-tuned baselines and produces coherent, high-quality reconstructions, all while significantly reducing inference cost. Code is available at https://github.com/YazidJanati/ding.