Anti-Overestimation Dialogue Policy Learning for Task-Completion Dialogue System
This work addresses a specific bottleneck in task-completion dialogue systems by mitigating overestimation in RL, offering an incremental improvement for developers of such systems.
The paper tackled the overestimation problem in reinforcement learning-based dialogue policy learning, which causes unstable training and suboptimal performance, by proposing a dynamic partial average estimator (DPAV) that adaptively weights predicted maximum and minimum action values. The method achieved better or comparable results to top baselines on three dialogue datasets with lower computational load, and included theoretical proofs of convergence and bias bounds.
A dialogue policy module is an essential part of task-completion dialogue systems. Recently, increasing interest has focused on reinforcement learning (RL)-based dialogue policy. Its favorable performance and wise action decisions rely on an accurate estimation of action values. The overestimation problem is a widely known issue of RL since its estimate of the maximum action value is larger than the ground truth, which results in an unstable learning process and suboptimal policy. This problem is detrimental to RL-based dialogue policy learning. To mitigate this problem, this paper proposes a dynamic partial average estimator (DPAV) of the ground truth maximum action value. DPAV calculates the partial average between the predicted maximum action value and minimum action value, where the weights are dynamically adaptive and problem-dependent. We incorporate DPAV into a deep Q-network as the dialogue policy and show that our method can achieve better or comparable results compared to top baselines on three dialogue datasets of different domains with a lower computational load. In addition, we also theoretically prove the convergence and derive the upper and lower bounds of the bias compared with those of other methods.