Diagnosing Transformers in Task-Oriented Semantic Parsing
This work addresses performance bottlenecks in semantic parsing for NLP applications, but it is incremental as it diagnoses existing models without proposing new solutions.
The study analyzed BART and XLM-R transformers in task-oriented semantic parsing, finding they struggle with disambiguating intents/slots and producing syntactically-valid frames, but provide indicators for frame correctness.
Modern task-oriented semantic parsing approaches typically use seq2seq transformers to map textual utterances to semantic frames comprised of intents and slots. While these models are empirically strong, their specific strengths and weaknesses have largely remained unexplored. In this work, we study BART and XLM-R, two state-of-the-art parsers, across both monolingual and multilingual settings. Our experiments yield several key results: transformer-based parsers struggle not only with disambiguating intents/slots, but surprisingly also with producing syntactically-valid frames. Though pre-training imbues transformers with syntactic inductive biases, we find the ambiguity of copying utterance spans into frames often leads to tree invalidity, indicating span extraction is a major bottleneck for current parsers. However, as a silver lining, we show transformer-based parsers give sufficient indicators for whether a frame is likely to be correct or incorrect, making them easier to deploy in production settings.