Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning
This work addresses performance issues in multi-domain task-oriented dialogue systems, which are incremental improvements for enhancing user task completion and response quality.
The paper tackled the exposure bias and token loss problems in task-oriented dialogue systems by optimizing a GPT2-based model with supervised and reinforcement learning using a non-differentiable reward function based on success rate and BLEU metrics. It achieved improvements of 1.60% in inform rate and 3.17% in success rate on the MultiWOZ2.1 dataset compared to a baseline.
Task-oriented dialogue (TOD) system is designed to accomplish user-defined tasks through dialogues. The TOD system has progressed towards end-to-end modeling by leveraging pre-trained large language models. Fine-tuning the pre-trained language models using only supervised learning leads to the exposure bias and token loss problem and it deviates the models from completing the user's task. To address these issues, we propose a TOD system that leverages a unified pre-trained language model, GPT2, as a base model. It is optimized using supervised learning and reinforcement learning (RL). The issues in the TOD system are mitigated using a non-differentiable reward function. The reward is calculated using the weighted sum of the success rate and BLEU evaluation metrics. The success rate and BLEU metrics in reward calculation guide the language model for user task completion while ensuring a coherent and fluent response. Our model is acquired by fine-tuning a pre-trained model on the dialogue-session level which comprises user utterance, belief state, system act, and system response. Experimental results on MultiWOZ2.1 demonstrate that our model increases the inform rate by 1.60% and the success rate by 3.17% compared to the baseline.