Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention
This work addresses the reliability of proactive interventions in AI agents, highlighting a critical gap in deployment safety for researchers and practitioners, though it is incremental in refining existing intervention frameworks.
The paper tackles the problem that accurate failure prediction in LLM critic models does not guarantee effective failure prevention, showing that a critic with high offline accuracy (AUROC 0.94) can cause severe performance degradation, such as a 26 percentage point collapse in one model, while having minimal effect on another.
Proactive interventions by LLM critic models are often assumed to improve reliability, yet their effects at deployment time are poorly understood. We show that a binary LLM critic with strong offline accuracy (AUROC 0.94) can nevertheless cause severe performance degradation, inducing a 26 percentage point (pp) collapse on one model while affecting another by near zero pp. This variability demonstrates that LLM critic accuracy alone is insufficient to determine whether intervention is safe. We identify a disruption-recovery tradeoff: interventions may recover failing trajectories but also disrupt trajectories that would have succeeded. Based on this insight, we propose a pre-deployment test that uses a small pilot of 50 tasks to estimate whether intervention is likely to help or harm, without requiring full deployment. Across benchmarks, the test correctly anticipates outcomes: intervention degrades performance on high-success tasks (0 to -26 pp), while yielding a modest improvement on the high-failure ALFWorld benchmark (+2.8 pp, p=0.014). The primary value of our framework is therefore identifying when not to intervene, preventing severe regressions before deployment.