Vision-Driven Prompt Optimization for Large Language Models in Multimodal Generative Tasks
This addresses the problem of integrating visual understanding and generative capabilities for researchers and practitioners in AI, offering a versatile solution for multimodal generative tasks, though it appears incremental as it builds on existing methods like LLMs and prompt tuning.
The paper tackles the challenge of vision generation by proposing Vision-Driven Prompt Optimization (VDPO), a framework that uses Large Language Models to generate textual prompts from visual inputs for image synthesis, achieving state-of-the-art performance with significant improvements in metrics like FID, LPIPS, and BLEU/CIDEr scores on benchmarks such as COCO and Sketchy.
Vision generation remains a challenging frontier in artificial intelligence, requiring seamless integration of visual understanding and generative capabilities. In this paper, we propose a novel framework, Vision-Driven Prompt Optimization (VDPO), that leverages Large Language Models (LLMs) to dynamically generate textual prompts from visual inputs, guiding high-fidelity image synthesis. VDPO combines a visual embedding prompt tuner, a textual instruction generator, and a vision generation module to achieve state-of-the-art performance in diverse vision generation tasks. Extensive experiments on benchmarks such as COCO and Sketchy demonstrate that VDPO consistently outperforms existing methods, achieving significant improvements in FID, LPIPS, and BLEU/CIDEr scores. Additional analyses reveal the scalability, robustness, and generalization capabilities of VDPO, making it a versatile solution for in-domain and out-of-domain tasks. Human evaluations further validate the practical superiority of VDPO in generating visually appealing and semantically coherent outputs.