WorkflowPerturb: Calibrated Stress Tests for Evaluating Multi-Agent Workflow Metrics
This addresses the problem of evaluating multi-agent workflow metrics for researchers and practitioners, though it is incremental as it provides a benchmark rather than a new evaluation method.
The paper tackles the problem of uncalibrated automatic evaluation metrics for LLM-generated structured workflows by introducing WorkflowPerturb, a benchmark with 4,973 golden workflows and 44,757 perturbed variants across three perturbation types at three severity levels. The results characterize systematic differences across metric families and enable severity-aware interpretation of workflow evaluation scores.
LLM-based systems increasingly generate structured workflows for complex tasks. In practice, automatic evaluation of these workflows is difficult, because metric scores are often not calibrated, and score changes do not directly communicate the severity of workflow degradation. We introduce WorkflowPerturb, a controlled benchmark for studying workflow evaluation metrics. It works by applying realistic, controlled perturbations to golden workflows. WorkflowPerturb contains 4,973 golden workflows and 44,757 perturbed variants across three perturbation types (Missing Steps, Compressed Steps, and Description Changes), each applied at severity levels of 10%, 30%, and 50%. We benchmark multiple metric families and analyze their sensitivity and calibration using expected score trajectories and residuals. Our results characterize systematic differences across metric families and support severity-aware interpretation of workflow evaluation scores. Our dataset will be released upon acceptance.