Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems
This work addresses the need for more realistic user simulators to enhance evaluation and data augmentation for task-oriented dialogue systems, representing an incremental improvement over existing methods.
The paper tackled the problem of unreliable user simulation in task-oriented dialogue systems by introducing DAUS, a domain-aware simulator fine-tuned on real dialogues, which improved user goal fulfillment on benchmarks and reduced hallucinations.
In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator's coherence with user goals, effectively mitigating hallucinations -- a major source of inconsistencies in simulator responses.