HAS-Bench: Evaluating LLM-Based Human-Agent Systems under Configurable Human Participation
Provides a benchmark for evaluating human-agent collaboration, addressing the need for systematic assessment of human participation in LLM systems.
HAS-Bench evaluates LLM-based human-agent systems under configurable human participation, finding that human input improves task completion and failure recovery, but gains depend on timing, method, and source of input.
Large language models increasingly operate in settings where humans are active collaborators rather than passive task providers. We introduce HAS-Framework, a graph-based framework that represents humans and LLM-powered agents as first-class participants with explicit roles, permissions, communication paths, and action authority. Building on this framework, HAS-Bench evaluates Human-Agent Systems under configurable human participation across agency levels, interaction channels, and persona policies. The benchmark measures both task outcomes and process-level collaboration behavior, including clarification quality, feedback utilization, control calibration, safety, initiative, and interaction cost. Experiments across six domains show that human participation can substantially improve task completion and failure recovery, but the gains depend on when, how, and by whom human input is exercised.