LUMINA: Long-horizon Understanding for Multi-turn Interactive Agents
This work addresses the challenge of improving AI agents for complex interactive tasks, but it is incremental as it focuses on diagnostic insights rather than a new solution.
The paper tackled the problem of understanding which underlying capabilities are most critical for AI agents to succeed in multi-turn, long-horizon tasks, using an oracle counterfactual framework to measure the impact of perfect skills like planning and state tracking, with results showing that some interventions consistently improve performance while others depend on environment and model properties.
Large language models can perform well on many isolated tasks, yet they continue to struggle on multi-turn, long-horizon agentic problems that require skills such as planning, state tracking, and long context processing. In this work, we aim to better understand the relative importance of advancing these underlying capabilities for success on such tasks. We develop an oracle counterfactual framework for multi-turn problems that asks: how would an agent perform if it could leverage an oracle to perfectly perform a specific task? The change in the agent's performance due to this oracle assistance allows us to measure the criticality of such oracle skill in the future advancement of AI agents. We introduce a suite of procedurally generated, game-like tasks with tunable complexity. These controlled environments allow us to provide precise oracle interventions, such as perfect planning or flawless state tracking, and make it possible to isolate the contribution of each oracle without confounding effects present in real-world benchmarks. Our results show that while some interventions (e.g., planning) consistently improve performance across settings, the usefulness of other skills is dependent on the properties of the environment and language model. Our work sheds light on the challenges of multi-turn agentic environments to guide the future efforts in the development of AI agents and language models.