ConvGenVisMo: Evaluation of Conversational Generative Vision Models
This work addresses the need for standardized evaluation in the emerging field of CGVMs, providing tools for researchers and practitioners to assess model performance, though it is incremental as it builds on existing evaluation concepts.
The paper tackles the problem of evaluating conversational generative vision models (CGVMs) by introducing ConvGenVisMo, a framework that includes a new benchmark dataset and a suite of automated evaluation metrics, with all assets made publicly available.
Conversational generative vision models (CGVMs) like Visual ChatGPT (Wu et al., 2023) have recently emerged from the synthesis of computer vision and natural language processing techniques. These models enable more natural and interactive communication between humans and machines, because they can understand verbal inputs from users and generate responses in natural language along with visual outputs. To make informed decisions about the usage and deployment of these models, it is important to analyze their performance through a suitable evaluation framework on realistic datasets. In this paper, we present ConvGenVisMo, a framework for the novel task of evaluating CGVMs. ConvGenVisMo introduces a new benchmark evaluation dataset for this task, and also provides a suite of existing and new automated evaluation metrics to evaluate the outputs. All ConvGenVisMo assets, including the dataset and the evaluation code, will be made available publicly on GitHub.