PATHWAYS: Evaluating Investigation and Context Discovery in AI Web Agents
This work addresses the problem of unreliable investigation and context discovery in AI web agents, which is incremental as it benchmarks existing architectures.
The authors introduced PATHWAYS, a benchmark of 250 multi-step decision tasks to test web-based agents' ability to discover and use hidden contextual information, finding that agents retrieved decisive evidence in only a small fraction of cases and performance dropped to near chance when overturning misleading signals.
We introduce PATHWAYS, a benchmark of 250 multi-step decision tasks that test whether web-based agents can discover and correctly use hidden contextual information. Across both closed and open models, agents typically navigate to relevant pages but retrieve decisive hidden evidence in only a small fraction of cases. When tasks require overturning misleading surface-level signals, performance drops sharply to near chance accuracy. Agents frequently hallucinate investigative reasoning by claiming to rely on evidence they never accessed. Even when correct context is discovered, agents often fail to integrate it into their final decision. Providing more explicit instructions improves context discovery but often reduces overall accuracy, revealing a tradeoff between procedural compliance and effective judgement. Together, these results show that current web agent architectures lack reliable mechanisms for adaptive investigation, evidence integration, and judgement override.