Concept Tagging for Natural Language Understanding: Two Decadelong Algorithm Development
This work provides a comprehensive resource for researchers in natural language understanding, but it is incremental as it reviews existing methods without introducing new ones.
The paper reviews and compares generative, discriminative, and deep learning methods for concept tagging in natural language understanding over 25 years, reporting statistical variability in performance on two public datasets and releasing a repository of algorithms and datasets.
Concept tagging is a type of structured learning needed for natural language understanding (NLU) systems. In this task, meaning labels from a domain ontology are assigned to word sequences. In this paper, we review the algorithms developed over the last twenty five years. We perform a comparative evaluation of generative, discriminative and deep learning methods on two public datasets. We report on the statistical variability performance measurements. The third contribution is the release of a repository of the algorithms, datasets and recipes for NLU evaluation.