Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search
This addresses the need for safe and efficient policy updates in high-risk applications like industrial control, though it appears incremental as it builds on existing batch reinforcement learning methods.
The paper tackles the problem of improving policies in batch reinforcement learning for industrial control by proposing an iterative approach that guides data collection during deployment, aiming for continuous policy improvement while staying within data support.
Batch reinforcement learning enables policy learning without direct interaction with the environment during training, relying exclusively on previously collected sets of interactions. This approach is, therefore, well-suited for high-risk and cost-intensive applications, such as industrial control. Learned policies are commonly restricted to act in a similar fashion as observed in the batch. In a real-world scenario, learned policies are deployed in the industrial system, inevitably leading to the collection of new data that can subsequently be added to the existing recording. The process of learning and deployment can thus take place multiple times throughout the lifespan of a system. In this work, we propose to exploit this iterative nature of applying offline reinforcement learning to guide learned policies towards efficient and informative data collection during deployment, leading to continuous improvement of learned policies while remaining within the support of collected data. We present an algorithmic methodology for iterative batch reinforcement learning based on ensemble-based model-based policy search, augmented with safety and, importantly, a diversity criterion.