Rethinking Failure Attribution in Multi-Agent Systems: A Multi-Perspective Benchmark and Evaluation
This work addresses the challenge of diagnosing failures in multi-agent systems for developers and researchers, offering a more realistic approach, though it is incremental in improving benchmark design.
The paper tackles the problem of failure attribution in multi-agent systems by proposing a multi-perspective paradigm to address attribution ambiguity, and introduces MP-Bench, a new benchmark and evaluation protocol, showing that prior conclusions about LLMs struggling with failure attribution are due to limitations in existing benchmarks.
Failure attribution is essential for diagnosing and improving multi-agent systems (MAS), yet existing benchmarks and methods largely assume a single deterministic root cause for each failure. In practice, MAS failures often admit multiple plausible attributions due to complex inter-agent dependencies and ambiguous execution trajectories. We revisit MAS failure attribution from a multi-perspective standpoint and propose multi-perspective failure attribution, a practical paradigm that explicitly accounts for attribution ambiguity. To support this setting, we introduce MP-Bench, the first benchmark designed for multi-perspective failure attribution in MAS, along with a new evaluation protocol tailored to this paradigm. Through extensive experiments, we find that prior conclusions suggesting LLMs struggle with failure attribution are largely driven by limitations in existing benchmark designs. Our results highlight the necessity of multi-perspective benchmarks and evaluation protocols for realistic and reliable MAS debugging.