AgentRx: Diagnosing AI Agent Failures from Execution Trajectories
This addresses the challenge of localizing failures in probabilistic, long-horizon AI agents for developers and researchers, though it is incremental as it builds on existing diagnostic methods.
The paper tackles the problem of diagnosing AI agent failures by introducing a benchmark of 115 annotated failed trajectories and an automated framework, AGENTRX, which improves step localization and failure attribution over existing baselines across domains.
AI agents often fail in ways that are difficult to localize because executions are probabilistic, long-horizon, multi-agent, and mediated by noisy tool outputs. We address this gap by manually annotating failed agent runs and release a novel benchmark of 115 failed trajectories spanning structured API workflows, incident management, and open-ended web/file tasks. Each trajectory is annotated with a critical failure step and a category from a grounded-theory derived, cross-domain failure taxonomy. To mitigate the human cost of failure attribution, we present AGENTRX, an automated domain-agnostic diagnostic framework that pinpoints the critical failure step in a failed agent trajectory. It synthesizes constraints, evaluates them step-by-step, and produces an auditable validation log of constraint violations with associated evidence; an LLM-based judge uses this log to localize the critical step and category. Our framework improves step localization and failure attribution over existing baselines across three domains.