Paragraph-to-Image Generation with Information-Enriched Diffusion Model
This work addresses the challenge of long-text alignment in image generation for applications requiring detailed scene depiction, representing an incremental advancement by fine-tuning existing models with new data and techniques.
The paper tackles the problem of generating images from long paragraphs, where existing text-to-image models struggle with alignment and complex scenes, by introducing ParaDiffusion, an information-enriched diffusion model that leverages large language models and a curated dataset, resulting in up to 15% and 45% improvements in human voting rates for visual appeal and text faithfulness compared to state-of-the-art models.
Text-to-image (T2I) models have recently experienced rapid development, achieving astonishing performance in terms of fidelity and textual alignment capabilities. However, given a long paragraph (up to 512 words), these generation models still struggle to achieve strong alignment and are unable to generate images depicting complex scenes. In this paper, we introduce an information-enriched diffusion model for paragraph-to-image generation task, termed ParaDiffusion, which delves into the transference of the extensive semantic comprehension capabilities of large language models to the task of image generation. At its core is using a large language model (e.g., Llama V2) to encode long-form text, followed by fine-tuning with LORA to alignthe text-image feature spaces in the generation task. To facilitate the training of long-text semantic alignment, we also curated a high-quality paragraph-image pair dataset, namely ParaImage. This dataset contains a small amount of high-quality, meticulously annotated data, and a large-scale synthetic dataset with long text descriptions being generated using a vision-language model. Experiments demonstrate that ParaDiffusion outperforms state-of-the-art models (SD XL, DeepFloyd IF) on ViLG-300 and ParaPrompts, achieving up to 15% and 45% human voting rate improvements for visual appeal and text faithfulness, respectively. The code and dataset will be released to foster community research on long-text alignment.