Provable Defense against Backdoor Policies in Reinforcement Learning
This addresses a security threat in reinforcement learning for agents that cannot retrain policies, though it is incremental as it builds on existing subspace trigger assumptions.
The paper tackles the problem of defending against backdoor policies in reinforcement learning by sanitizing a policy through projecting observed states to a safe subspace, achieving ε approximate optimality with a provable sample complexity bound and demonstrating empirical success on Atari games.
We propose a provable defense mechanism against backdoor policies in reinforcement learning under subspace trigger assumption. A backdoor policy is a security threat where an adversary publishes a seemingly well-behaved policy which in fact allows hidden triggers. During deployment, the adversary can modify observed states in a particular way to trigger unexpected actions and harm the agent. We assume the agent does not have the resources to re-train a good policy. Instead, our defense mechanism sanitizes the backdoor policy by projecting observed states to a 'safe subspace', estimated from a small number of interactions with a clean (non-triggered) environment. Our sanitized policy achieves $ε$ approximate optimality in the presence of triggers, provided the number of clean interactions is $O\left(\frac{D}{(1-γ)^4 ε^2}\right)$ where $γ$ is the discounting factor and $D$ is the dimension of state space. Empirically, we show that our sanitization defense performs well on two Atari game environments.