Worrisome Properties of Neural Network Controllers and Their Symbolic Representations
This highlights a critical robustness problem in RL controllers for safety-critical applications, though it is incremental in focusing on specific benchmark scenarios.
The paper identifies that neural network controllers in reinforcement learning benchmarks, despite achieving high mean returns, often produce many persistent low-return solutions, which adversaries can exploit, with simpler controllers being more prone to this issue.
We raise concerns about controllers' robustness in simple reinforcement learning benchmark problems. We focus on neural network controllers and their low neuron and symbolic abstractions. A typical controller reaching high mean return values still generates an abundance of persistent low-return solutions, which is a highly undesirable property, easily exploitable by an adversary. We find that the simpler controllers admit more persistent bad solutions. We provide an algorithm for a systematic robustness study and prove existence of persistent solutions and, in some cases, periodic orbits, using a computer-assisted proof methodology.