The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation
This work addresses limitations in text-to-image generation for artistic and specialized prompts, offering a domain-specific solution.
The paper tackles the challenge of generating images from complex and imaginative prompts by introducing RFNet, a training-free method that integrates diffusion models with LLMs, achieving superiority over state-of-the-art methods in human and GPT-based evaluations.
In spite of recent advancements in text-to-image generation, limitations persist in handling complex and imaginative prompts due to the restricted diversity and complexity of training data. This work explores how diffusion models can generate images from prompts requiring artistic creativity or specialized knowledge. We introduce the Realistic-Fantasy Benchmark (RFBench), a novel evaluation framework blending realistic and fantastical scenarios. To address these challenges, we propose the Realistic-Fantasy Network (RFNet), a training-free approach integrating diffusion models with LLMs. Extensive human evaluations and GPT-based compositional assessments demonstrate our approach's superiority over state-of-the-art methods. Our code and dataset is available at https://leo81005.github.io/Reality-and-Fantasy/.