VisualPrompter: Prompt Optimization with Visual Feedback for Text-to-Image Synthesis
This addresses the issue of generating content-wise satisfying images from user prompts for users of diffusion models, representing an incremental improvement by focusing on semantic alignment.
The paper tackles the problem of semantic misalignment in text-to-image synthesis by proposing VisualPrompter, a training-free prompt optimization framework that refines user inputs to improve text-image alignment, achieving state-of-the-art performance on multiple benchmarks.
Since there exists a notable gap between user-provided and model-preferred prompts, generating high-quality and satisfactory images using diffusion models often requires prompt engineering to optimize user inputs. Current studies on text-to-image prompt engineering can effectively enhance the style and aesthetics of generated images. However, they often neglect the semantic alignment between generated images and user descriptions, resulting in visually appealing but content-wise unsatisfying outputs. In this work, we propose VisualPrompter, a novel training-free prompt engineering framework that refines user inputs to model-preferred sentences. In particular, VisualPrompter utilizes an automatic self-reflection module to identify the missing concepts in generated images and a target-specific prompt optimization mechanism to revise the prompts in a fine-grained manner. Extensive experiments demonstrate the effectiveness of our VisualPrompter, which achieves new state-of-the-art performance on multiple benchmarks for text-image alignment evaluation. Additionally, our framework features a plug-and-play design, making it highly adaptable to various generative models.