Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction
For AI researchers studying active concept induction, this work provides a benchmark and reveals specific weaknesses in current agents, particularly in active information gathering.
The paper introduces ZendoWorld, a visual concept induction environment, and evaluates various AI agents (VLM, Bayesian, neuro-symbolic) on it. Key findings: high label accuracy does not imply rule recovery, perception and induction are distinct bottlenecks, and VLM agents fail to design informative experiments, with humans outperforming agents on complex rules.
A central challenge in building intelligent systems is enabling agents to jointly perceive complex inputs, form hypotheses about hidden patterns, and design informative experiments to test them. To study this problem, we propose ZendoWorld, a controlled interactive environment in which agents must infer a logical rule about visual game observations, acquire information by proposing new scenes, and refine their hypotheses based on feedback from the game environment. We evaluate several agents spanning pure VLM reasoning, Bayesian particle filtering, dynamic concept discovery, and neuro-symbolic methods. Our main findings are: (1) high accuracy in predicting labels for observed examples does not imply recovery of the underlying rule; (2) perception and induction are distinct bottlenecks for different agent classes; and (3) VLM-based agents propose near-uninformative experiments, failing to actively reduce hypothesis uncertainty. To compare these results, we collect human data on the task, which reveals a gap in inductive reasoning, particularly for more complex rules. Overall, ZENDOWORLD takes an important step toward evaluating intelligent agents and identifies concrete avenues for improvement, particularly in domains like scientific discovery.