AMIGO: Agentic Multi-Image Grounding Oracle Benchmark
This addresses the problem of evaluating agentic models in extended, realistic scenarios for researchers, though it is incremental as it builds on existing evaluation frameworks.
The authors tackled the lack of benchmarks for agentic vision-language models in multi-image, multi-turn interactions by introducing AMIGO, a benchmark for hidden-target identification over galleries of similar images, which includes metrics for identification success and interaction quality.
Agentic vision-language models increasingly act through extended interactions, but most evaluations still focus on single-image, single-turn correctness. We introduce AMIGO (Agentic Multi-Image Grounding Oracle Benchmark), a long-horizon benchmark for hidden-target identification over galleries of visually similar images. In AMIGO, the oracle privately selects a target image, and the model must recover it by asking a sequence of attribute-focused Yes/No/Unsure questions under a strict protocol that penalizes invalid actions with Skip. This setting stresses (i) question selection under uncertainty, (ii) consistent constraint tracking across turns, and (iii) fine-grained discrimination as evidence accumulates. AMIGO also supports controlled oracle imperfections to probe robustness and verification behavior under inconsistent feedback. We instantiate AMIGO with Guess My Preferred Dress task and report metrics covering both outcomes and interaction quality, including identification success, evidence verification, efficiency, protocol compliance, noise tolerance, and trajectory-level diagnostics.