World-To-Image: Grounding Text-to-Image Generation with Agent-Driven World Knowledge
This addresses the limitation of text-to-image generation for handling real-world, evolving concepts, though it is an incremental improvement over existing methods.
The paper tackles the problem of text-to-image models degrading with novel or out-of-distribution entities by introducing World-To-Image, a framework that uses an agent to retrieve web images for unknown concepts and optimize prompts, resulting in an 8.1% improvement in accuracy-to-prompt on the NICE benchmark.
While text-to-image (T2I) models can synthesize high-quality images, their performance degrades significantly when prompted with novel or out-of-distribution (OOD) entities due to inherent knowledge cutoffs. We introduce World-To-Image, a novel framework that bridges this gap by empowering T2I generation with agent-driven world knowledge. We design an agent that dynamically searches the web to retrieve images for concepts unknown to the base model. This information is then used to perform multimodal prompt optimization, steering powerful generative backbones toward an accurate synthesis. Critically, our evaluation goes beyond traditional metrics, utilizing modern assessments like LLMGrader and ImageReward to measure true semantic fidelity. Our experiments show that World-To-Image substantially outperforms state-of-the-art methods in both semantic alignment and visual aesthetics, achieving +8.1% improvement in accuracy-to-prompt on our curated NICE benchmark. Our framework achieves these results with high efficiency in less than three iterations, paving the way for T2I systems that can better reflect the ever-changing real world. Our demo code is available here\footnote{https://github.com/mhson-kyle/World-To-Image}.