Text-Anchored Score Composition: Tackling Condition Misalignment in Text-to-Image Diffusion Models
This addresses a practical issue for users of controllable image generation by enhancing flexibility when conditions are partially aligned, though it is incremental as it builds on existing models.
The paper tackles the problem of condition misalignment in text-to-image diffusion models, where imperfect alignment between text and extra conditions leads to ambiguous or dominated outputs, by introducing a training-free method called Text-Anchored Score Composition (TASC) that improves controllability and performs favorably against recent methods.
Text-to-image diffusion models have advanced towards more controllable generation via supporting various additional conditions (e.g.,depth map, bounding box) beyond text. However, these models are learned based on the premise of perfect alignment between the text and extra conditions. If this alignment is not satisfied, the final output could be either dominated by one condition, or ambiguity may arise, failing to meet user expectations. To address this issue, we present a training free approach called Text-Anchored Score Composition (TASC) to further improve the controllability of existing models when provided with partially aligned conditions. The TASC firstly separates conditions based on pair relationships, computing the result individually for each pair. This ensures that each pair no longer has conflicting conditions. Then we propose an attention realignment operation to realign these independently calculated results via a cross-attention mechanism to avoid new conflicts when combining them back. Both qualitative and quantitative results demonstrate the effectiveness of our approach in handling unaligned conditions, which performs favorably against recent methods and more importantly adds flexibility to the controllable image generation process. Our code will be available at: https://github.com/EnVision-Research/Decompose-and-Realign.