REALab: An Embedded Perspective on Tampering
This work is significant for AI safety researchers and practitioners concerned with the potential for RL agents to maliciously alter their reward signals in real-world embedded systems.
This paper introduces REALab, a platform for studying embedded agency in reinforcement learning (RL) where agents can tamper with their feedback mechanisms. It addresses the unrealistic assumption of secure feedback in standard RL by proposing a Corrupt Feedback MDP formulation and a corresponding environment platform.
This paper describes REALab, a platform for embedded agency research in reinforcement learning (RL). REALab is designed to model the structure of tampering problems that may arise in real-world deployments of RL. Standard Markov Decision Process (MDP) formulations of RL and simulated environments mirroring the MDP structure assume secure access to feedback (e.g., rewards). This may be unrealistic in settings where agents are embedded and can corrupt the processes producing feedback (e.g., human supervisors, or an implemented reward function). We describe an alternative Corrupt Feedback MDP formulation and the REALab environment platform, which both avoid the secure feedback assumption. We hope the design of REALab provides a useful perspective on tampering problems, and that the platform may serve as a unit test for the presence of tampering incentives in RL agent designs.