Multi-Faceted Evaluation of Tool-Augmented Dialogue Systems
This addresses a key challenge in tool-augmented dialogue systems for AI developers, offering a more robust evaluation method, though it is incremental as it builds on existing evaluation approaches.
The paper tackles the problem of evaluating conversational AI systems that use external tools, where existing methods fail to capture critical errors like misinterpretation of tool results, and introduces TRACE, a benchmark for synthesized conversations, and SCOPE, an evaluation framework that outperforms baselines, especially in challenging cases with misleading user satisfaction signals.
Evaluating conversational AI systems that use external tools is challenging, as errors can arise from complex interactions among user, agent, and tools. While existing evaluation methods assess either user satisfaction or agents' tool-calling capabilities, they fail to capture critical errors in multi-turn tool-augmented dialogues-such as when agents misinterpret tool results yet appear satisfactory to users. We introduce TRACE, a benchmark of systematically synthesized tool-augmented conversations covering diverse error cases, and SCOPE, an evaluation framework that automatically discovers diverse error patterns and evaluation rubrics in tool-augmented dialogues. Experiments show SCOPE significantly outperforms the baseline, particularly on challenging cases where user satisfaction signals are misleading.