Robust Adversarial Reinforcement Learning in Stochastic Games via Sequence Modeling
This work addresses adversarial robustness for reinforcement learning practitioners, representing an incremental advance by adapting sequence modeling to adversarial settings.
The paper tackles the problem of adversarial robustness in reinforcement learning methods based on sequence modeling, specifically the Decision Transformer, by introducing the Conservative Adversarially Robust Decision Transformer (CART), which achieves more accurate minimax value estimation and superior worst-case returns in adversarial stochastic games.
The Transformer, a highly expressive architecture for sequence modeling, has recently been adapted to solve sequential decision-making, most notably through the Decision Transformer (DT), which learns policies by conditioning on desired returns. Yet, the adversarial robustness of reinforcement learning methods based on sequence modeling remains largely unexplored. Here we introduce the Conservative Adversarially Robust Decision Transformer (CART), to our knowledge the first framework designed to enhance the robustness of DT in adversarial stochastic games. We formulate the interaction between the protagonist and the adversary at each stage as a stage game, where the payoff is defined as the expected maximum value over subsequent states, thereby explicitly incorporating stochastic state transitions. By conditioning Transformer policies on the NashQ value derived from these stage games, CART generates policy that are simultaneously less exploitable (adversarially robust) and conservative to transition uncertainty. Empirically, CART achieves more accurate minimax value estimation and consistently attains superior worst-case returns across a range of adversarial stochastic games.