Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation
This provides a scalable solution for e-commerce platforms to evaluate retrieval systems, though it is incremental as it applies existing LLM capabilities to a specific domain.
The paper tackles the challenge of evaluating product retrieval systems at scale by proposing a framework that uses multimodal LLMs to generate annotation guidelines and perform annotations, achieving comparable quality to human annotations while reducing time and cost.
Evaluating production-level retrieval systems at scale is a crucial yet challenging task due to the limited availability of a large pool of well-trained human annotators. Large Language Models (LLMs) have the potential to address this scaling issue and offer a viable alternative to humans for the bulk of annotation tasks. In this paper, we propose a framework for assessing the product search engines in a large-scale e-commerce setting, leveraging Multimodal LLMs for (i) generating tailored annotation guidelines for individual queries, and (ii) conducting the subsequent annotation task. Our method, validated through deployment on a large e-commerce platform, demonstrates comparable quality to human annotations, significantly reduces time and cost, facilitates rapid problem discovery, and provides an effective solution for production-level quality control at scale.