Rescue Conversations from Dead-ends: Efficient Exploration for Task-oriented Dialogue Policy Optimization
This addresses inefficiencies in dialogue policy optimization for task-oriented systems, offering an incremental improvement over existing methods.
The paper tackles the problem of inefficient exploration in task-oriented dialogue policy training by identifying dead-end states as a key cause of wasted exploration, and proposes a dead-end resurrection algorithm that detects these states and provides rescue actions to improve learning efficiency, achieving experimental results across multiple dialogue datasets.
Training a dialogue policy using deep reinforcement learning requires a lot of exploration of the environment. The amount of wasted invalid exploration makes their learning inefficient. In this paper, we find and define an important reason for the invalid exploration: dead-ends. When a conversation enters a dead-end state, regardless of the actions taken afterward, it will continue in a dead-end trajectory until the agent reaches a termination state or maximum turn. We propose a dead-end resurrection (DDR) algorithm that detects the initial dead-end state in a timely and efficient manner and provides a rescue action to guide and correct the exploration direction. To prevent dialogue policies from repeatedly making the same mistake, DDR also performs dialogue data augmentation by adding relevant experiences containing dead-end states. We first validate the dead-end detection reliability and then demonstrate the effectiveness and generality of the method by reporting experimental results on several dialogue datasets from different domains.