Generative Induction of Dialogue Task Schemas with Streaming Refinement and Simulated Interactions
This work advances automated dialogue understanding for developers, though it builds incrementally on existing LLM-based methods.
The paper tackles Slot Schema Induction (SSI) for task-oriented dialogue systems by formulating it as a text generation task, achieving state-of-the-art results through incremental refinement over dialogue streams. It also addresses evaluation issues by creating new data with human guidance and designing improved metrics.
In task-oriented dialogue (TOD) systems, Slot Schema Induction (SSI) is essential for automatically identifying key information slots from dialogue data without manual intervention. This paper presents a novel state-of-the-art (SoTA) approach that formulates SSI as a text generation task, where a language model incrementally constructs and refines a slot schema over a stream of dialogue data. To develop this approach, we present a fully automatic LLM-based TOD simulation method that creates data with high-quality state labels for novel task domains. Furthermore, we identify issues in SSI evaluation due to data leakage and poor metric alignment with human judgment. We resolve these by creating new evaluation data using our simulation method with human guidance and correction, as well as designing improved evaluation metrics. These contributions establish a foundation for future SSI research and advance the SoTA in dialogue understanding and system development.