LocateBench: Evaluating the Locating Ability of Vision Language Models
This work addresses the need for better evaluation of vision language models in object localization tasks, which is important for real-world applications, but it is incremental as it focuses on benchmarking rather than model improvement.
The authors tackled the problem of evaluating vision language models' ability to locate objects in images by proposing LocateBench, a new benchmark, and found that the strongest model, GPT-4o, had an accuracy more than 10% lower than human performance.
The ability to locate an object in an image according to natural language instructions is crucial for many real-world applications. In this work we propose LocateBench, a high-quality benchmark dedicated to evaluating this ability. We experiment with multiple prompting approaches, and measure the accuracy of several large vision language models. We find that even the accuracy of the strongest model, GPT-4o, lags behind human accuracy by more than 10%.