Dominating vs. Dominated: Generative Collapse in Diffusion Models
This addresses a reliability issue in text-to-image generation for users, but it is incremental as it analyzes an existing problem without proposing a new solution.
The paper tackles the problem of one concept token dominating others in multi-concept text-to-image diffusion models, termed Dominant-vs-Dominated imbalance, and finds that limited instance diversity in training data and cross-attention dynamics cause this generative collapse.
Text-to-image diffusion models have drawn significant attention for their ability to generate diverse and high-fidelity images. However, when generating from multi-concept prompts, one concept token often dominates the generation, suppressing the others-a phenomenon we term the Dominant-vs-Dominated (DvD) imbalance. To systematically analyze this imbalance, we introduce DominanceBench and examine its causes from both data and architectural perspectives. Through various experiments, we show that the limited instance diversity in training data exacerbates the inter-concept interference. Analysis of cross-attention dynamics further reveals that dominant tokens rapidly saturate attention, progressively suppressing others across diffusion timesteps. In addition, head ablation studies show that the DvD behavior arises from distributed attention mechanisms across multiple heads. Our findings provide key insights into generative collapse, advancing toward more reliable and controllable text-to-image generation.