SOP-Maze: Evaluating Large Language Models on Complicated Business Standard Operating Procedures
This work addresses a gap in evaluating LLMs for domain-specific agents in business contexts, though it is incremental as it builds on existing benchmarking efforts.
The authors tackled the problem of evaluating large language models (LLMs) on complex business standard operating procedures (SOPs) by proposing SOP-Maze, a benchmark with 397 tasks from 23 real-world scenarios, and found that nearly all state-of-the-art models struggle with it, identifying key error categories like route blindness and conversational fragility.
As large language models (LLMs) are widely deployed as domain-specific agents, many benchmarks have been proposed to evaluate their ability to follow instructions and make decisions in real-world scenarios. However, business scenarios often involve complex standard operating procedures (SOPs), and the evaluation of LLM capabilities in such contexts has not been fully explored. To bridge this gap, we propose SOP-Maze, a benchmark constructed from real-world business data and adapted into a collection of 397 tasks from 23 complex SOP scenarios. We further categorize SOP tasks into two broad classes: Lateral Root System (LRS), representing wide-option tasks that demand precise selection; and Heart Root System (HRS), which emphasizes deep logical reasoning with complex branches. Extensive experiments reveal that nearly all state-of-the-art models struggle with SOP-Maze. We conduct a comprehensive analysis and identify three key error categories: (i) route blindness: difficulty following procedures; (ii) conversational fragility: inability to handle real dialogue nuances; and (iii) calculation errors: mistakes in time or arithmetic reasoning under complex contexts. The systematic study explores LLM performance across SOP tasks that challenge both breadth and depth, offering new insights for improving model capabilities. We have open-sourced our work on https://github.com/ADoublLEN/SOP-Maze.