Refine-by-Align: Reference-Guided Artifacts Refinement through Semantic Alignment
This addresses the problem of fine-grained identity loss in generated images for users of personalized image generation models, representing a novel approach in a previously underexplored area.
The paper tackles localized artifacts like incorrect logos in personalized image generation by introducing a reference-guided artifacts refinement task and proposing Refine-by-Align, a diffusion-based model that improves image fidelity and identity details, generalizing across various tasks without test-time tuning.
Personalized image generation has emerged from the recent advancements in generative models. However, these generated personalized images often suffer from localized artifacts such as incorrect logos, reducing fidelity and fine-grained identity details of the generated results. Furthermore, there is little prior work tackling this problem. To help improve these identity details in the personalized image generation, we introduce a new task: reference-guided artifacts refinement. We present Refine-by-Align, a first-of-its-kind model that employs a diffusion-based framework to address this challenge. Our model consists of two stages: Alignment Stage and Refinement Stage, which share weights of a unified neural network model. Given a generated image, a masked artifact region, and a reference image, the alignment stage identifies and extracts the corresponding regional features in the reference, which are then used by the refinement stage to fix the artifacts. Our model-agnostic pipeline requires no test-time tuning or optimization. It automatically enhances image fidelity and reference identity in the generated image, generalizing well to existing models on various tasks including but not limited to customization, generative compositing, view synthesis, and virtual try-on. Extensive experiments and comparisons demonstrate that our pipeline greatly pushes the boundary of fine details in the image synthesis models.