Sample-Efficient Model-based Actor-Critic for an Interactive Dialogue Task
This addresses the data scarcity issue for developers of human-computer interactive systems like digital assistants, though it is incremental as it builds on existing actor-critic methods.
The paper tackles the problem of sample inefficiency in interactive dialogue tasks by proposing a model-based reinforcement learning algorithm, which requires 70 times fewer samples and achieves 2 times better asymptotic performance compared to a model-free baseline.
Human-computer interactive systems that rely on machine learning are becoming paramount to the lives of millions of people who use digital assistants on a daily basis. Yet, further advances are limited by the availability of data and the cost of acquiring new samples. One way to address this problem is by improving the sample efficiency of current approaches. As a solution path, we present a model-based reinforcement learning algorithm for an interactive dialogue task. We build on commonly used actor-critic methods, adding an environment model and planner that augments a learning agent to learn the model of the environment dynamics. Our results show that, on a simulation that mimics the interactive task, our algorithm requires 70 times fewer samples, compared to the baseline of commonly used model-free algorithm, and demonstrates 2~times better performance asymptotically. Moreover, we introduce a novel contribution of computing a soft planner policy and further updating a model-free policy yielding a less computationally expensive model-free agent as good as the model-based one. This model-based architecture serves as a foundation that can be extended to other human-computer interactive tasks allowing further advances in this direction.