CRAB-Bench: Evaluating LLM Agents under Complex Task Dependencies and Human-aligned User Simulation
For researchers evaluating LLM agents in service scenarios, this benchmark reveals that current agents struggle with realistic user behavior, especially information disclosure, and tend to mask errors.
CRAB-Bench introduces a benchmark for LLM agents with complex task dependencies and a realistic user simulator (RUSE) based on human behavioral studies. The best agent achieves only 61% pass@1, and RUSE causes performance drops of up to 57%.
Evaluating LLM agents in realistic service scenarios requires complex task dependencies, imperfect user behavior, and an evaluation that accommodates multiple valid solutions. We introduce CRAB-Bench (Constraint-based Realistic Agent Benchmark) and RUSE (Realistic User Simulation Engine) to address this gap. CRAB-Bench generates tasks via a constraint graph over multiple interdependent entities with structured distractors, requiring agents to reason carefully over thousands of misleading candidates where only a tiny fraction of solutions are valid. RUSE replaces cooperative, template-like simulators with realistic users grounded in human behavioral studies, instantiated across diverse personas and four behavioral dimensions. Experiments on four frontier LLM agents show that the best model achieves only 61% pass@1 on CRAB-Bench, and switching to RUSE causes further drops of up to 57%, concentrated in task-solving ability rather than conversational quality. Information Disclosure is the most damaging behavioral dimension, and agents interacting with RUSE are less likely to admit mistakes, instead masking errors through implicit corrections.