Dynamic Shielding for Reinforcement Learning in Black-Box Environments
This addresses safety concerns in RL for cyber-physical systems, offering a novel approach that is incremental by extending existing shielding techniques.
The paper tackled the problem of ensuring safety in reinforcement learning for cyber-physical systems without prior system knowledge by proposing dynamic shielding, which constructs an approximate model using automata learning to suppress unsafe actions, resulting in a significant decrease in undesired events during training.
It is challenging to use reinforcement learning (RL) in cyber-physical systems due to the lack of safety guarantees during learning. Although there have been various proposals to reduce undesired behaviors during learning, most of these techniques require prior system knowledge, and their applicability is limited. This paper aims to reduce undesired behaviors during learning without requiring any prior system knowledge. We propose dynamic shielding: an extension of a model-based safe RL technique called shielding using automata learning. The dynamic shielding technique constructs an approximate system model in parallel with RL using a variant of the RPNI algorithm and suppresses undesired explorations due to the shield constructed from the learned model. Through this combination, potentially unsafe actions can be foreseen before the agent experiences them. Experiments show that our dynamic shield significantly decreases the number of undesired events during training.