Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models
This work addresses the need for more reliable evaluation metrics in text-to-image generation, which is important for researchers and developers in AI and creative fields, though it is incremental as it builds on existing multimodal methods.
The paper tackles the problem of unreliable assessment of text-to-image models due to cross-modal information asymmetry by introducing an Image Regeneration task that uses GPT4V to bridge image and text, enabling evaluation through direct image comparisons, and demonstrates that this approach allows models to generate images more closely resembling reference images.
Diffusion models have revitalized the image generation domain, playing crucial roles in both academic research and artistic expression. With the emergence of new diffusion models, assessing the performance of text-to-image models has become increasingly important. Current metrics focus on directly matching the input text with the generated image, but due to cross-modal information asymmetry, this leads to unreliable or incomplete assessment results. Motivated by this, we introduce the Image Regeneration task in this study to assess text-to-image models by tasking the T2I model with generating an image according to the reference image. We use GPT4V to bridge the gap between the reference image and the text input for the T2I model, allowing T2I models to understand image content. This evaluation process is simplified as comparisons between the generated image and the reference image are straightforward. Two regeneration datasets spanning content-diverse and style-diverse evaluation dataset are introduced to evaluate the leading diffusion models currently available. Additionally, we present ImageRepainter framework to enhance the quality of generated images by improving content comprehension via MLLM guided iterative generation and revision. Our comprehensive experiments have showcased the effectiveness of this framework in assessing the generative capabilities of models. By leveraging MLLM, we have demonstrated that a robust T2M can produce images more closely resembling the reference image.