Improving Robustness of Task Oriented Dialog Systems
This addresses robustness issues in task-oriented dialog systems, which are critical for real-world applications like virtual assistants, but the approach is incremental as it adapts existing adversarial methods from computer vision to language tasks.
The paper tackles the problem of task-oriented dialog systems being overly sensitive to small input variations, which reduces their accuracy and reliability. The authors create an adversarial test set to measure robustness and show that adversarial training and back-translation data augmentation substantially improve robustness, with combined methods yielding the best results.
Task oriented language understanding in dialog systems is often modeled using intents (task of a query) and slots (parameters for that task). Intent detection and slot tagging are, in turn, modeled using sentence classification and word tagging techniques respectively. Similar to adversarial attack problems with computer vision models discussed in existing literature, these intent-slot tagging models are often over-sensitive to small variations in input -- predicting different and often incorrect labels when small changes are made to a query, thus reducing their accuracy and reliability. However, evaluating a model's robustness to these changes is harder for language since words are discrete and an automated change (e.g. adding `noise') to a query sometimes changes the meaning and thus labels of a query. In this paper, we first describe how to create an adversarial test set to measure the robustness of these models. Furthermore, we introduce and adapt adversarial training methods as well as data augmentation using back-translation to mitigate these issues. Our experiments show that both techniques improve the robustness of the system substantially and can be combined to yield the best results.