Approaching Dialogue State Tracking via Aligning Speech Encoders and LLMs
This work addresses spoken dialogue understanding for conversational AI systems, representing an incremental improvement with specific gains.
The paper tackled spoken Dialogue State Tracking by aligning speech encoders and LLMs with a connector module, achieving state-of-the-art results of 34.66% JGA on SpokenWOZ test with their best model and 42.17% JGA with a Gemma-2-9B-instruct variant.
In this work, we approach spoken Dialogue State Tracking (DST) by bridging the representation spaces of speech encoders and LLMs via a small connector module, with a focus on fully open-sourced and open-data components (WavLM-large, OLMo). We focus on ablating different aspects of such systems including full/LoRA adapter fine-tuning, the effect of agent turns in the dialogue history, as well as fuzzy matching-based output post-processing, which greatly improves performance of our systems on named entities in the dialogue slot values. We conduct our experiments on the SpokenWOZ dataset, and additionally utilize the Speech-Aware MultiWOZ dataset to augment our training data. Ultimately, our best-performing WavLM + connector + OLMo-1B aligned models achieve state of the art on the SpokenWOZ test set (34.66% JGA), and our system with Gemma-2-9B-instruct further surpasses this result, reaching 42.17% JGA on SpokenWOZ test.