Adaptive Auxiliary Prompt Blending for Target-Faithful Diffusion Generation
This addresses a domain-specific issue for users of text-to-image generation by improving faithfulness in low-density regions, though it is incremental as it builds on existing prompt blending methods.
The paper tackles the problem of diffusion-based text-to-image models producing misaligned results for rare concepts by introducing Adaptive Auxiliary Prompt Blending (AAPB), which uses adaptive coefficients to blend prompts and achieves superior semantic accuracy and structural fidelity on RareBench and FlowEdit datasets.
Diffusion-based text-to-image (T2I) models have made remarkable progress in generating photorealistic and semantically rich images. However, when the target concepts lie in low-density regions of the training distribution, these models often produce semantically misaligned or structurally inconsistent results. This limitation arises from the long-tailed nature of text-image datasets, where rare concepts or editing instructions are underrepresented. To address this, we introduce Adaptive Auxiliary Prompt Blending (AAPB) - a unified framework that stabilizes the diffusion process in low-density regions. AAPB leverages auxiliary anchor prompts to provide semantic support in rare concept generation and structural support in image editing, ensuring faithful guidance toward the target prompt. Unlike prior heuristic prompt alternation methods, AAPB derives a closed-form adaptive coefficient that optimally balances the influence between the auxiliary anchor and the target prompt at each diffusion step. Grounded in Tweedie's identity, our formulation provides a principled and training-free framework for adaptive prompt blending, ensuring stable and target-faithful generation. We demonstrate the effectiveness of adaptive interpolation over fixed interpolation through controlled experiments and empirically show consistent improvements on the RareBench and FlowEdit datasets, achieving superior semantic accuracy and structural fidelity compared to prior training-free baselines.