Generalizing from a few environments in safety-critical reinforcement learning
This addresses safety-critical deployment of autonomous agents, but the results are incremental as they show limited generalization beyond simple environments.
The paper tackles the problem of ensuring safety in reinforcement learning agents when they encounter novel environments, finding that simple modifications like ensemble averaging can reduce catastrophes in gridworlds but not in more complex settings, though ensemble uncertainty can predict imminent catastrophes to trigger human intervention.
Before deploying autonomous agents in the real world, we need to be confident they will perform safely in novel situations. Ideally, we would expose agents to a very wide range of situations during training, allowing them to learn about every possible danger, but this is often impractical. This paper investigates safety and generalization from a limited number of training environments in deep reinforcement learning (RL). We find RL algorithms can fail dangerously on unseen test environments even when performing perfectly on training environments. Firstly, in a gridworld setting, we show that catastrophes can be significantly reduced with simple modifications, including ensemble model averaging and the use of a blocking classifier. In the more challenging CoinRun environment we find similar methods do not significantly reduce catastrophes. However, we do find that the uncertainty information from the ensemble is useful for predicting whether a catastrophe will occur within a few steps and hence whether human intervention should be requested.