Grounding Language Models for Visual Entity Recognition
This work addresses the challenge of recognizing visual entities in images for applications like multimodal AI, though it appears incremental as it builds on existing autoregressive models with constrained generation.
The paper tackles the problem of visual entity recognition by introducing AutoVER, an autoregressive model that improves accuracy on out-of-domain entities and visually-situated reasoning. It achieves a significant increase in accuracy from 32.7% to 61.5% on the Entity seen split of the Oven-Wiki benchmark.
We introduce AutoVER, an Autoregressive model for Visual Entity Recognition. Our model extends an autoregressive Multi-modal Large Language Model by employing retrieval augmented constrained generation. It mitigates low performance on out-of-domain entities while excelling in queries that require visually-situated reasoning. Our method learns to distinguish similar entities within a vast label space by contrastively training on hard negative pairs in parallel with a sequence-to-sequence objective without an external retriever. During inference, a list of retrieved candidate answers explicitly guides language generation by removing invalid decoding paths. The proposed method achieves significant improvements across different dataset splits in the recently proposed Oven-Wiki benchmark. Accuracy on the Entity seen split rises from 32.7% to 61.5%. It also demonstrates superior performance on the unseen and query splits by a substantial double-digit margin.