Correcting Diffusion Generation through Resampling
This addresses image quality and object fidelity issues in text-to-image generation, representing an incremental improvement over existing methods.
The paper tackles distributional discrepancies in diffusion models that cause missing objects and low image quality in image generation by proposing a particle filtering framework that uses external guidance (real images and object detectors) to correct these gaps. The method improves object occurrence by 5% and FID by 1.0 on MS-COCO compared to baselines.
Despite diffusion models' superior capabilities in modeling complex distributions, there are still non-trivial distributional discrepancies between generated and ground-truth images, which has resulted in several notable problems in image generation, including missing object errors in text-to-image generation and low image quality. Existing methods that attempt to address these problems mostly do not tend to address the fundamental cause behind these problems, which is the distributional discrepancies, and hence achieve sub-optimal results. In this paper, we propose a particle filtering framework that can effectively address both problems by explicitly reducing the distributional discrepancies. Specifically, our method relies on a set of external guidance, including a small set of real images and a pre-trained object detector, to gauge the distribution gap, and then design the resampling weight accordingly to correct the gap. Experiments show that our methods can effectively correct missing object errors and improve image quality in various image generation tasks. Notably, our method outperforms the existing strongest baseline by 5% in object occurrence and 1.0 in FID on MS-COCO. Our code is publicly available at https://github.com/UCSB-NLP-Chang/diffusion_resampling.git.