Avoiding Tampering Incentives in Deep RL via Decoupled Approval
This work is significant for designers of AI systems, as it tackles the fundamental problem of ensuring agents pursue their intended objectives without manipulating their feedback mechanisms, a critical concern for safe and robust AI deployment.
This paper addresses the problem of deep reinforcement learning agents tampering with their reward mechanisms by proposing a decoupled approval approach. This method ensures aligned incentives for agents even when feedback is influenceable, both at convergence and during local updates, and is shown to scale to complex 3D environments.
How can we design agents that pursue a given objective when all feedback mechanisms are influenceable by the agent? Standard RL algorithms assume a secure reward function, and can thus perform poorly in settings where agents can tamper with the reward-generating mechanism. We present a principled solution to the problem of learning from influenceable feedback, which combines approval with a decoupled feedback collection procedure. For a natural class of corruption functions, decoupled approval algorithms have aligned incentives both at convergence and for their local updates. Empirically, they also scale to complex 3D environments where tampering is possible.