DeepStress: Stress-Testing Deep Search Agents
For developers of search agents, this work highlights a critical but under-explored failure mode (poor-quality evidence) and provides a controlled evaluation method, though it is incremental in applying synthetic environments to known robustness concerns.
DeepStress introduces a stress-testing framework that controls evidence reliability (trustworthiness, relevance, factuality) to evaluate search agents' robustness. Testing on HotpotQA and BrowseCompPlus reveals substantial performance differences among agents and proposes new metrics for documenting system outcomes.
While search agents demonstrate impressive capabilities in multi-step question answering, their robustness to poor-quality evidence remains under-explored. This phenomenon occurs rarely in realistic benchmarks but can lead to dramatic failure in real life applications. Therefore in this study we propose DeepStress, a stress testing framework that controls the frequency of challenging evidence by replacing the retrieval module of search agents with a controlled synthetic environment. We use this framework to control three dimensions that can affect document reliability: trustworthiness, relevance, and factuality. Testing several search agents on HotpotQA and BrowseCompPlus, we demonstrate that agents exhibit substantial differences in their ability to handle unreliable information and propose new metrics that better document systems outcomes as well as the interactions between conflicting parametric and retrieved knowledge.