NATURE: Natural Auxiliary Text Utterances for Realistic Spoken Language Evaluation
This addresses the gap between benchmark performance and real-world generalization for conversational agents, though it is incremental as it focuses on evaluation rather than new model development.
The authors tackled the problem of evaluating slot-filling and intent detection models in realistic spoken language scenarios by introducing NATURE, a set of transformations that add human-like variations to benchmark datasets while preserving semantics. They demonstrated that applying NATURE to evaluation sets can cause model accuracy to drop by up to 40%.
Slot-filling and intent detection are the backbone of conversational agents such as voice assistants, and are active areas of research. Even though state-of-the-art techniques on publicly available benchmarks show impressive performance, their ability to generalize to realistic scenarios is yet to be demonstrated. In this work, we present NATURE, a set of simple spoken-language oriented transformations, applied to the evaluation set of datasets, to introduce human spoken language variations while preserving the semantics of an utterance. We apply NATURE to common slot-filling and intent detection benchmarks and demonstrate that simple perturbations from the standard evaluation set by NATURE can deteriorate model performance significantly. Through our experiments we demonstrate that when NATURE operators are applied to evaluation set of popular benchmarks the model accuracy can drop by up to 40%.