Dialogue Generation: From Imitation Learning to Inverse Reinforcement Learning
This addresses a specific bottleneck in dialogue generation for conversational AI systems, representing an incremental improvement over existing adversarial approaches.
The paper tackles the problem of sparse and unstable reward signals in adversarial dialogue generation by proposing a new reward model within an adversarial inverse reinforcement learning framework. Experimental results show the model generates higher-quality responses and achieves better overall performance than state-of-the-art methods.
The performance of adversarial dialogue generation models relies on the quality of the reward signal produced by the discriminator. The reward signal from a poor discriminator can be very sparse and unstable, which may lead the generator to fall into a local optimum or to produce nonsense replies. To alleviate the first problem, we first extend a recently proposed adversarial dialogue generation method to an adversarial imitation learning solution. Then, in the framework of adversarial inverse reinforcement learning, we propose a new reward model for dialogue generation that can provide a more accurate and precise reward signal for generator training. We evaluate the performance of the resulting model with automatic metrics and human evaluations in two annotation settings. Our experimental results demonstrate that our model can generate more high-quality responses and achieve higher overall performance than the state-of-the-art.