Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach
This addresses the challenge of lacking clean, large-scale training data for associating images with entities in knowledge bases like Wikipedia, which is crucial for improving visual recognition systems.
The paper tackled the problem of web-scale visual entity recognition by curating a high-quality training dataset using a multimodal LLM for label verification and enrichment, resulting in models that achieve state-of-the-art performance with a +6.9% improvement on the OVEN entity task.
Web-scale visual entity recognition, the task of associating images with their corresponding entities within vast knowledge bases like Wikipedia, presents significant challenges due to the lack of clean, large-scale training data. In this paper, we propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Instead of relying on the multimodal LLM to directly annotate data, which we found to be suboptimal, we prompt it to reason about potential candidate entity labels by accessing additional contextually relevant information (such as Wikipedia), resulting in more accurate annotations. We further use the multimodal LLM to enrich the dataset by generating question-answer pairs and a grounded finegrained textual description (referred to as "rationale") that explains the connection between images and their assigned entities. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks (e.g. +6.9% improvement in OVEN entity task), underscoring the importance of high-quality training data in this domain.