ECom-Bench: Can LLM Agent Resolve Real-World E-commerce Customer Support Issues?
This addresses the problem of evaluating AI agents for e-commerce customer support, providing a standardized benchmark for researchers and developers, though it is incremental as it focuses on benchmarking rather than new methods.
The paper introduces ECom-Bench, a benchmark framework for evaluating multimodal LLM agents in e-commerce customer support, where even advanced models like GPT-4o achieve only 10-20% pass^3 metric, highlighting the challenge of real-world complexities.
In this paper, we introduce ECom-Bench, the first benchmark framework for evaluating LLM agent with multimodal capabilities in the e-commerce customer support domain. ECom-Bench features dynamic user simulation based on persona information collected from real e-commerce customer interactions and a realistic task dataset derived from authentic e-commerce dialogues. These tasks, covering a wide range of business scenarios, are designed to reflect real-world complexities, making ECom-Bench highly challenging. For instance, even advanced models like GPT-4o achieve only a 10-20% pass^3 metric in our benchmark, highlighting the substantial difficulties posed by complex e-commerce scenarios. The code and data have been made publicly available at https://github.com/XiaoduoAILab/ECom-Bench to facilitate further research and development in this domain.