DreamSampler: Unifying Diffusion Sampling and Score Distillation for Image Manipulation
This work addresses mode-collapsing and architectural constraints in image manipulation for users of latent diffusion models, offering a more flexible and robust tool, though it appears incremental as it builds on existing sampling and distillation methods.
The paper tackles the limitations of reverse diffusion sampling and score-distillation for image manipulation in latent diffusion models by introducing DreamSampler, a model-agnostic framework that integrates both approaches through regularized latent optimization, demonstrating competitive performance in tasks like image editing and SVG reconstruction.
Reverse sampling and score-distillation have emerged as main workhorses in recent years for image manipulation using latent diffusion models (LDMs). While reverse diffusion sampling often requires adjustments of LDM architecture or feature engineering, score distillation offers a simple yet powerful model-agnostic approach, but it is often prone to mode-collapsing. To address these limitations and leverage the strengths of both approaches, here we introduce a novel framework called {\em DreamSampler}, which seamlessly integrates these two distinct approaches through the lens of regularized latent optimization. Similar to score-distillation, DreamSampler is a model-agnostic approach applicable to any LDM architecture, but it allows both distillation and reverse sampling with additional guidance for image editing and reconstruction. Through experiments involving image editing, SVG reconstruction and etc, we demonstrate the competitive performance of DreamSampler compared to existing approaches, while providing new applications. Code: https://github.com/DreamSampler/dream-sampler