SafeLife 1.0: Exploring Side Effects in Complex Environments
This work addresses the safety of reinforcement learning agents in dynamic environments, though it is incremental as it provides a benchmark without novel safety solutions.
The authors tackled the problem of evaluating safety in reinforcement learning agents by introducing SafeLife, a complex environment that tests for side effects, and found that agents trained with proximal policy optimization achieved high performance but caused significant side effects, establishing a baseline for future safety research.
We present SafeLife, a publicly available reinforcement learning environment that tests the safety of reinforcement learning agents. It contains complex, dynamic, tunable, procedurally generated levels with many opportunities for unsafe behavior. Agents are graded both on their ability to maximize their explicit reward and on their ability to operate safely without unnecessary side effects. We train agents to maximize rewards using proximal policy optimization and score them on a suite of benchmark levels. The resulting agents are performant but not safe -- they tend to cause large side effects in their environments -- but they form a baseline against which future safety research can be measured.