Refining Visual Artifacts in Diffusion Models via Explainable AI-based Flaw Activation Maps
This work addresses image quality issues in diffusion models, which is crucial for applications in image generation and synthesis, representing an incremental advancement by integrating explainable AI for refinement.
The paper tackles the problem of artifacts and unrealistic regions in diffusion models for image synthesis by proposing a self-refining framework that uses explainable AI-based flaw activation maps to detect and correct these flaws, achieving up to a 27.3% improvement in Fréchet inception distance across various models and tasks.
Diffusion models have achieved remarkable success in image synthesis. However, addressing artifacts and unrealistic regions remains a critical challenge. We propose self-refining diffusion, a novel framework that enhances image generation quality by detecting these flaws. The framework employs an explainable artificial intelligence (XAI)-based flaw highlighter to produce flaw activation maps (FAMs) that identify artifacts and unrealistic regions. These FAMs improve reconstruction quality by amplifying noise in flawed regions during the forward process and by focusing on these regions during the reverse process. The proposed approach achieves up to a 27.3% improvement in Fréchet inception distance across various diffusion-based models, demonstrating consistently strong performance on diverse datasets. It also shows robust effectiveness across different tasks, including image generation, text-to-image generation, and inpainting. These results demonstrate that explainable AI techniques can extend beyond interpretability to actively contribute to image refinement. The proposed framework offers a versatile and effective approach applicable to various diffusion models and tasks, significantly advancing the field of image synthesis.