Learning to Teach Reinforcement Learning Agents
This addresses the challenge of transfer learning for heterogeneous students in reinforcement learning, but it is incremental as it builds on existing advice models with specific improvements.
The paper tackles the problem of efficiently providing action advice to reinforcement learning agents under a limited budget, focusing on Pac-Man, and shows that using the coefficient of variation as a statistic improves advice quality, while proposing a novel RL algorithm for learning when to advise.
In this article we study the transfer learning model of action advice under a budget. We focus on reinforcement learning teachers providing action advice to heterogeneous students playing the game of Pac-Man under a limited advice budget. First, we examine several critical factors affecting advice quality in this setting, such as the average performance of the teacher, its variance and the importance of reward discounting in advising. The experiments show the non-trivial importance of the coefficient of variation (CV) as a statistic for choosing policies that generate advice. The CV statistic relates variance to the corresponding mean. Second, the article studies policy learning for distributing advice under a budget. Whereas most methods in the relevant literature rely on heuristics for advice distribution we formulate the problem as a learning one and propose a novel RL algorithm capable of learning when to advise, adapting to the student and the task at hand. Furthermore, we argue that learning to advise under a budget is an instance of a more generic learning problem: Constrained Exploitation Reinforcement Learning.