OWLViz: An Open-World Benchmark for Visual Question Answering
This work addresses the challenge of practical AI systems for multimodal tasks, establishing a new benchmark to reveal limitations in tool selection and reasoning, though it is incremental as it builds on existing VLM research.
The paper tackles the problem of open-world visual question answering by introducing the OWLViz benchmark, which requires integrating visual understanding, web exploration, and tool usage, and finds that while humans achieve 69.2% accuracy, the best model (Gemini 2.0) only achieves 26.6%, highlighting a large performance gap.
We present a challenging benchmark for the Open WorLd VISual question answering (OWLViz) task. OWLViz presents concise, unambiguous queries that require integrating multiple capabilities, including visual understanding, web exploration, and specialized tool usage. While humans achieve 69.2% accuracy on these intuitive tasks, even state-of-the-art VLMs struggle, with the best model, Gemini 2.0, achieving only 26.6% accuracy. Current agentic VLMs, which rely on limited vision and vision-language models as tools, perform even worse. This performance gap reveals significant limitations in multimodal systems' ability to select appropriate tools and execute complex reasoning sequences, establishing new directions for advancing practical AI research.