ChatChecker: A Framework for Dialogue System Testing and Evaluation Through Non-cooperative User Simulation
This work addresses the problem of integrated dialogue-level quality assurance for researchers and practitioners developing robust dialogue systems, representing an incremental advancement with novel components like non-cooperative user simulation.
The authors tackled the challenge of testing and evaluating complex dialogue systems as a whole, beyond just the underlying LLMs, by introducing ChatChecker, a framework that uses LLM-based user simulation to detect dialogue breakdowns and improve detection performance over prior methods.
While modern dialogue systems heavily rely on large language models (LLMs), their implementation often goes beyond pure LLM interaction. Developers integrate multiple LLMs, external tools, and databases. Therefore, assessment of the underlying LLM alone does not suffice, and the dialogue systems must be tested and evaluated as a whole. However, this remains a major challenge. With most previous work focusing on turn-level analysis, less attention has been paid to integrated dialogue-level quality assurance. To address this, we present ChatChecker, a framework for automated evaluation and testing of complex dialogue systems. ChatChecker uses LLMs to simulate diverse user interactions, identify dialogue breakdowns, and evaluate quality. Compared to previous approaches, our design reduces setup effort and is generalizable, as it does not require reference dialogues and is decoupled from the implementation of the target dialogue system. We improve breakdown detection performance over a prior LLM-based approach by including an error taxonomy in the prompt. Additionally, we propose a novel non-cooperative user simulator based on challenging personas that uncovers weaknesses in target dialogue systems more effectively. Through this, ChatChecker contributes to thorough and scalable testing. This enables both researchers and practitioners to accelerate the development of robust dialogue systems.