LumiGen: An LVLM-Enhanced Iterative Framework for Fine-Grained Text-to-Image Generation
This work addresses challenges in text-to-image generation for applications requiring precise control, such as accurate text rendering and pose generation, representing a novel integration of LVLMs but with incremental improvements in specific areas.
The paper tackles the problem of fine-grained control and semantic consistency in text-to-image generation by proposing LumiGen, an LVLM-enhanced iterative framework that uses a closed-loop feedback mechanism, achieving a superior average score of 3.08 on the LongBench-T2I Benchmark and outperforming state-of-the-art baselines.
Text-to-Image (T2I) generation has made significant advancements with diffusion models, yet challenges persist in handling complex instructions, ensuring fine-grained content control, and maintaining deep semantic consistency. Existing T2I models often struggle with tasks like accurate text rendering, precise pose generation, or intricate compositional coherence. Concurrently, Vision-Language Models (LVLMs) have demonstrated powerful capabilities in cross-modal understanding and instruction following. We propose LumiGen, a novel LVLM-enhanced iterative framework designed to elevate T2I model performance, particularly in areas requiring fine-grained control, through a closed-loop, LVLM-driven feedback mechanism. LumiGen comprises an Intelligent Prompt Parsing & Augmentation (IPPA) module for proactive prompt enhancement and an Iterative Visual Feedback & Refinement (IVFR) module, which acts as a "visual critic" to iteratively correct and optimize generated images. Evaluated on the challenging LongBench-T2I Benchmark, LumiGen achieves a superior average score of 3.08, outperforming state-of-the-art baselines. Notably, our framework demonstrates significant improvements in critical dimensions such as text rendering and pose expression, validating the effectiveness of LVLM integration for more controllable and higher-quality image generation.