Non-Cooperative Inverse Reinforcement Learning
This addresses the challenge of inferring objectives in adversarial settings, such as cyber security, but appears incremental as it builds upon existing multi-agent IRL frameworks.
The paper tackles the problem of decision-making against a strategic opponent who can hide their true objective, by introducing the Non-Cooperative Inverse Reinforcement Learning (N-CIRL) formalism as a zero-sum Markov game with one-sided incomplete information. It develops a computationally tractable algorithm to solve for equilibrium strategies and demonstrates benefits over existing methods in a cyber security simulation.
Making decisions in the presence of a strategic opponent requires one to take into account the opponent's ability to actively mask its intended objective. To describe such strategic situations, we introduce the non-cooperative inverse reinforcement learning (N-CIRL) formalism. The N-CIRL formalism consists of two agents with completely misaligned objectives, where only one of the agents knows the true objective function. Formally, we model the N-CIRL formalism as a zero-sum Markov game with one-sided incomplete information. Through interacting with the more informed player, the less informed player attempts to both infer, and act according to, the true objective function. As a result of the one-sided incomplete information, the multi-stage game can be decomposed into a sequence of single-stage games expressed by a recursive formula. Solving this recursive formula yields the value of the N-CIRL game and the more informed player's equilibrium strategy. Another recursive formula, constructed by forming an auxiliary game, termed the dual game, yields the less informed player's strategy. Building upon these two recursive formulas, we develop a computationally tractable algorithm to approximately solve for the equilibrium strategies. Finally, we demonstrate the benefits of our N-CIRL formalism over the existing multi-agent IRL formalism via extensive numerical simulation in a novel cyber security setting.