Deceptive Reinforcement Learning Under Adversarial Manipulations on Cost Signals
This addresses security risks in learning-based control systems, but it is incremental as it builds on existing RL frameworks with a focus on adversarial robustness.
The paper tackles the problem of reinforcement learning vulnerabilities under adversarial cost signal manipulation, showing that Q-learning converges under stealthy attacks and establishing fundamental limits for feasible offensive and defensive moves, with a numerical case study on water reservoir control illustrating potential hazards.
This paper studies reinforcement learning (RL) under malicious falsification on cost signals and introduces a quantitative framework of attack models to understand the vulnerabilities of RL. Focusing on $Q$-learning, we show that $Q$-learning algorithms converge under stealthy attacks and bounded falsifications on cost signals. We characterize the relation between the falsified cost and the $Q$-factors as well as the policy learned by the learning agent which provides fundamental limits for feasible offensive and defensive moves. We propose a robust region in terms of the cost within which the adversary can never achieve the targeted policy. We provide conditions on the falsified cost which can mislead the agent to learn an adversary's favored policy. A numerical case study of water reservoir control is provided to show the potential hazards of RL in learning-based control systems and corroborate the results.