Mitigating Planner Overfitting in Model-Based Reinforcement Learning
This addresses a key challenge for reinforcement learning agents by improving robustness against model inaccuracies, though it appears incremental as it builds on existing regularization concepts.
The paper tackles the problem of planner overfitting in model-based reinforcement learning, where agents exploit model errors, and presents three approaches that effectively mitigate this issue in reinforcement-learning environments.
An agent with an inaccurate model of its environment faces a difficult choice: it can ignore the errors in its model and act in the real world in whatever way it determines is optimal with respect to its model. Alternatively, it can take a more conservative stance and eschew its model in favor of optimizing its behavior solely via real-world interaction. This latter approach can be exceedingly slow to learn from experience, while the former can lead to "planner overfitting" - aspects of the agent's behavior are optimized to exploit errors in its model. This paper explores an intermediate position in which the planner seeks to avoid overfitting through a kind of regularization of the plans it considers. We present three different approaches that demonstrably mitigate planner overfitting in reinforcement-learning environments.