TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
This addresses the problem of inadequate evaluation for advanced visual reasoning in AI research, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the lack of benchmarks for advanced agentic thinking-with-images reasoning by introducing TIR-Bench, a comprehensive benchmark with 13 diverse tasks, and found it universally challenging for 22 evaluated multimodal large language models, with strong performance requiring genuine capabilities.
The frontier of visual reasoning is shifting toward models like OpenAI o3, which can intelligently create and operate tools to transform images for problem-solving, also known as thinking-\textit{with}-images in chain-of-thought. Yet existing benchmarks fail to fully capture this advanced capability. Even Visual Search, the most common benchmark for current thinking-\textit{with}-images methods, tests only basic operations such as localization and cropping, offering little insight into more complex, dynamic, and tool-dependent reasoning. We introduce \textbf{TIR-Bench}, a comprehensive benchmark for evaluating agentic thinking-with-images across 13 diverse tasks, each requiring novel tool use for image processing and manipulation in chain-of-thought. We evaluate 22 multimodal large language models (MLLMs), from leading open-sourced and proprietary models to those with explicit tool-use augmentation. Results show that TIR-Bench is universally challenging, and strong performance requires genuine thinking-with-images capabilities. Finally, we present a pilot study comparing direct versus agentic fine-tuning.