What Do LLM Agents Know About Their World? Task2Quiz: A Paradigm for Studying Environment Understanding
This addresses a critical gap in evaluating LLM agents' generalization for developing more robust autonomous systems, though it is incremental in providing a new evaluation method rather than a solution.
The paper tackled the problem of evaluating whether LLM agents truly understand their environments, beyond just task success, by proposing the Task-to-Quiz (T2Q) paradigm and T2QBench with 30 environments and 1,967 QA pairs. The results showed that task success is a poor proxy for environment understanding and identified bottlenecks like proactive exploration and state representation.
Large language model (LLM) agents have demonstrated remarkable capabilities in complex decision-making and tool-use tasks, yet their ability to generalize across varying environments remains a under-examined concern. Current evaluation paradigms predominantly rely on trajectory-based metrics that measure task success, while failing to assess whether agents possess a grounded, transferable model of the environment. To address this gap, we propose Task-to-Quiz (T2Q), a deterministic and automated evaluation paradigm designed to decouple task execution from world-state understanding. We instantiate this paradigm in T2QBench, a suite comprising 30 environments and 1,967 grounded QA pairs across multiple difficulty levels. Our extensive experiments reveal that task success is often a poor proxy for environment understanding, and that current memory machanism can not effectively help agents acquire a grounded model of the environment. These findings identify proactive exploration and fine-grained state representation as primary bottlenecks, offering a robust foundation for developing more generalizable autonomous agents.