Conformal Intent Classification and Clarification for Fast and Accurate Intent Recognition
This work addresses the challenge of improving user experience in dialogue agents by providing specific clarification questions, though it is incremental as it builds on existing intent classifiers and uncertainty methods.
The paper tackles the problem of fast and accurate intent classification in task-oriented dialogue systems by introducing Conformal Intent Classification and Clarification (CICC), which uses uncertainty scores to generate clarification questions guaranteed to contain the true intent at a pre-defined confidence level, resulting in small clarification questions and out-of-scope detection capabilities across seven datasets.
We present Conformal Intent Classification and Clarification (CICC), a framework for fast and accurate intent classification for task-oriented dialogue systems. The framework turns heuristic uncertainty scores of any intent classifier into a clarification question that is guaranteed to contain the true intent at a pre-defined confidence level. By disambiguating between a small number of likely intents, the user query can be resolved quickly and accurately. Additionally, we propose to augment the framework for out-of-scope detection. In a comparative evaluation using seven intent recognition datasets we find that CICC generates small clarification questions and is capable of out-of-scope detection. CICC can help practitioners and researchers substantially in improving the user experience of dialogue agents with specific clarification questions.