FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL
This addresses the problem of precise visual control in text-to-image generation for users needing fine-grained semantic alignment, representing a strong incremental improvement over existing autoregressive approaches.
The paper tackles the problem of poor fine-grained text-image alignment in autoregressive text-to-image models, which struggle with subtle semantic differences between similar prompts. The proposed FocusDiff method achieves state-of-the-art performance on existing benchmarks and significantly outperforms prior methods on their new PairComp benchmark.
Recent studies extend the autoregression paradigm to text-to-image generation, achieving performance comparable to diffusion models. However, our new PairComp benchmark -- featuring test cases of paired prompts with similar syntax but different fine-grained semantics -- reveals that existing models struggle with fine-grained text-image alignment thus failing to realize precise control over visual tokens. To address this, we propose FocusDiff, which enhances fine-grained text-image semantic alignment by focusing on subtle differences between similar text-image pairs. We construct a new dataset of paired texts and images with similar overall expressions but distinct local semantics, further introducing a novel reinforcement learning algorithm to emphasize such fine-grained semantic differences for desired image generation. Our approach achieves state-of-the-art performance on existing text-to-image benchmarks and significantly outperforms prior methods on PairComp.