Adapting Long Context NLM for ASR Rescoring in Conversational Agents
This work addresses the problem of enhancing ASR accuracy and downstream natural language understanding for conversational agents, representing an incremental improvement with specific domain applications.
The paper tackled improving automatic speech recognition (ASR) rescoring in conversational agents by adapting neural language models (NLMs) to use long context from multiple utterances, resulting in a relative word error rate reduction of 1.6% to 9.1% and a 4% improvement in slot labeling F1 score over non-contextual baselines.
Neural Language Models (NLM), when trained and evaluated with context spanning multiple utterances, have been shown to consistently outperform both conventional n-gram language models and NLMs that use limited context. In this paper, we investigate various techniques to incorporate turn based context history into both recurrent (LSTM) and Transformer-XL based NLMs. For recurrent based NLMs, we explore context carry over mechanism and feature based augmentation, where we incorporate other forms of contextual information such as bot response and system dialogue acts as classified by a Natural Language Understanding (NLU) model. To mitigate the sharp nearby, fuzzy far away problem with contextual NLM, we propose the use of attention layer over lexical metadata to improve feature based augmentation. Additionally, we adapt our contextual NLM towards user provided on-the-fly speech patterns by leveraging encodings from a large pre-trained masked language model and performing fusion with a Transformer-XL based NLM. We test our proposed models using N-best rescoring of ASR hypotheses of task-oriented dialogues and also evaluate on downstream NLU tasks such as intent classification and slot labeling. The best performing model shows a relative WER between 1.6% and 9.1% and a slot labeling F1 score improvement of 4% over non-contextual baselines.