LLaVA-RE: Binary Image-Text Relevancy Evaluation with Multimodal Large Language Model
This work addresses a fundamental challenge in multimodal AI evaluation, though it is incremental as it builds on existing LLaVA architecture.
The paper tackles the problem of binary image-text relevancy evaluation by introducing LLaVA-RE, a framework based on a Multimodal Large Language Model, which effectively handles diverse text formats and varying relevancy definitions across tasks.
Multimodal generative AI usually involves generating image or text responses given inputs in another modality. The evaluation of image-text relevancy is essential for measuring response quality or ranking candidate responses. In particular, binary relevancy evaluation, i.e., ``Relevant'' vs. ``Not Relevant'', is a fundamental problem. However, this is a challenging task considering that texts have diverse formats and the definition of relevancy varies in different scenarios. We find that Multimodal Large Language Models (MLLMs) are an ideal choice to build such evaluators, as they can flexibly handle complex text formats and take in additional task information. In this paper, we present LLaVA-RE, a first attempt for binary image-text relevancy evaluation with MLLM. It follows the LLaVA architecture and adopts detailed task instructions and multimodal in-context samples. In addition, we propose a novel binary relevancy data set that covers various tasks. Experimental results validate the effectiveness of our framework.