Stealthy Imitation: Reward-guided Environment-free Policy Stealing
This addresses a security vulnerability for policy owners in control systems, but it is incremental as it builds on existing model stealing concepts.
The paper tackles the problem of stealing deep reinforcement learning policies without environment access by proposing Stealthy Imitation, which fits a reward model to approximate the victim's input distribution, and shows it outperforms prior data-free methods across high-dimensional control tasks.
Deep reinforcement learning policies, which are integral to modern control systems, represent valuable intellectual property. The development of these policies demands considerable resources, such as domain expertise, simulation fidelity, and real-world validation. These policies are potentially vulnerable to model stealing attacks, which aim to replicate their functionality using only black-box access. In this paper, we propose Stealthy Imitation, the first attack designed to steal policies without access to the environment or knowledge of the input range. This setup has not been considered by previous model stealing methods. Lacking access to the victim's input states distribution, Stealthy Imitation fits a reward model that allows to approximate it. We show that the victim policy is harder to imitate when the distribution of the attack queries matches that of the victim. We evaluate our approach across diverse, high-dimensional control tasks and consistently outperform prior data-free approaches adapted for policy stealing. Lastly, we propose a countermeasure that significantly diminishes the effectiveness of the attack.