Device Directedness with Contextual Cues for Spoken Dialog Systems
This work addresses the challenge of improving response accuracy and speed in spoken dialog systems for users, though it is incremental as it builds on existing pre-trained models.
The paper tackles the problem of verifying barge-in in spoken dialog systems by defining it as a supervised learning task using audio-only information, and proposes a model that infuses lexical information into speech representations, achieving up to 5.7% relative F1 score improvement and being 22% faster than a baseline.
In this work, we define barge-in verification as a supervised learning task where audio-only information is used to classify user spoken dialogue into true and false barge-ins. Following the success of pre-trained models, we use low-level speech representations from a self-supervised representation learning model for our downstream classification task. Further, we propose a novel technique to infuse lexical information directly into speech representations to improve the domain-specific language information implicitly learned during pre-training. Experiments conducted on spoken dialog data show that our proposed model trained to validate barge-in entirely from speech representations is faster by 38% relative and achieves 4.5% relative F1 score improvement over a baseline LSTM model that uses both audio and Automatic Speech Recognition (ASR) 1-best hypotheses. On top of this, our best proposed model with lexically infused representations along with contextual features provides a further relative improvement of 5.7% in the F1 score but only 22% faster than the baseline.