Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers
This work addresses the problem of prompt inversion for image generation models, providing a comparative analysis that is incremental in nature.
The study compared discrete optimization methods for recovering natural language prompts from images generated by AI models, finding that while optimizers effectively minimize their objectives, using a trained captioner often produces images more similar to those from the original prompts.
Recovering natural language prompts for image generation models, solely based on the generated images is a difficult discrete optimization problem. In this work, we present the first head-to-head comparison of recent discrete optimization techniques for the problem of prompt inversion. We evaluate Greedy Coordinate Gradients (GCG), PEZ , Random Search, AutoDAN and BLIP2's image captioner across various evaluation metrics related to the quality of inverted prompts and the quality of the images generated by the inverted prompts. We find that focusing on the CLIP similarity between the inverted prompts and the ground truth image acts as a poor proxy for the similarity between ground truth image and the image generated by the inverted prompts. While the discrete optimizers effectively minimize their objectives, simply using responses from a well-trained captioner often leads to generated images that more closely resemble those produced by the original prompts.