CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environments
This addresses the need for more realistic benchmarks in AI-driven customer service, though it is incremental as it builds on existing evaluation methods by adding efficiency metrics.
The authors tackled the problem of evaluating LLM-based agents in real-world cloud service environments by introducing CirrusBench, a framework based on authentic customer service tickets, which revealed that state-of-the-art models often struggle with complex multi-turn tasks and fail to meet efficiency standards, with metrics like Normalized Efficiency Index showing significant gaps.
The increasing agentic capabilities of Large Language Models (LLMs) have enabled their deployment in real-world applications, such as cloud services, where customer-assistant interactions exhibit high technical complexity and long-horizon dependencies, making robustness and resolution efficiency critical for customer satisfaction. However, existing benchmarks for LLM-based agents largely rely on synthetic environments that fail to capture the diversity and unpredictability of authentic customer inputs, often ignoring the resolution efficiency essential for real-world deployment. To bridge this gap, we introduce CirrusBench, a novel evaluation framework distinguished by its foundation in real-world data from authentic cloud service tickets. CirrusBench preserves the intricate multi-turn logical chains and realistic tool dependencies inherent to technical service environments. Moving beyond execution correctness, we introduce novel Customer-Centric metrics to define agent success, quantifying service quality through metrics such as the Normalized Efficiency Index and Multi-Turn Latency to explicitly measure resolution efficiency. Experiments utilizing our framework reveal that while state-of-the-art models demonstrate strong reasoning capabilities, they frequently struggle in complex, realistic multi-turn tasks and fail to meet the high-efficiency standards required for customer service, highlighting critical directions for the future development of LLM-based agents in practical technical service applications. CirrusBench evaluation framework is released at: https://github.com/CirrusAI