Secure Reinforcement Learning: On Model-Free Detection of Man in the Middle Attacks
For cyber-physical systems using RL, this work provides a provably optimal detection scheme against sophisticated MITM attacks, addressing a critical security bottleneck.
This paper extends the Bellman Deviation Detection framework to detect man-in-the-middle attacks in model-free reinforcement learning, proving that the agent's detection time scales linearly with the adversary's learning time, which is order-optimal.
We consider the problem of learning-based man-in-the-middle (MITM) attacks in cyber-physical systems (CPS), and extend our previously proposed Bellman Deviation Detection (BDD) framework for model-free reinforcement learning (RL). We refine the standard MDP attack model by allowing the reward function to depend on both the current and subsequent states, thereby capturing reward variations induced by errors in the adversary's transition estimate. We also derive an optimal system-identification strategy for the adversary that minimizes detectable value deviations. Further, we prove that the agent's asymptotic learning time required to secure the system scales linearly with the adversary's learning time, and that this matches the optimal lower bound. Hence, the proposed detection scheme is order-optimal in detection efficiency. Finally, we extend the framework to asynchronous and intermittent attack scenarios, where reliable detection is preserved.