PoolFlip: A Multi-Agent Reinforcement Learning Security Environment for Cyber Defense
This work addresses cyber defense automation for security professionals, but it is incremental as it builds on the existing FlipIt framework with new learning techniques.
The paper tackles the problem of automating cyber defense against stealthy adversaries by introducing PoolFlip, a multi-agent reinforcement learning environment based on the FlipIt game, and Flip-PSRO, a method that trains defenders to generalize to unknown attacks, resulting in defenders being 2× more effective than baselines.
Cyber defense requires automating defensive decision-making under stealthy, deceptive, and continuously evolving adversarial strategies. The FlipIt game provides a foundational framework for modeling interactions between a defender and an advanced adversary that compromises a system without being immediately detected. In FlipIt, the attacker and defender compete to control a shared resource by performing a Flip action and paying a cost. However, the existing FlipIt frameworks rely on a small number of heuristics or specialized learning techniques, which can lead to brittleness and the inability to adapt to new attacks. To address these limitations, we introduce PoolFlip, a multi-agent gym environment that extends the FlipIt game to allow efficient learning for attackers and defenders. Furthermore, we propose Flip-PSRO, a multi-agent reinforcement learning (MARL) approach that leverages population-based training to train defender agents equipped to generalize against a range of unknown, potentially adaptive opponents. Our empirical results suggest that Flip-PSRO defenders are $2\times$ more effective than baselines to generalize to a heuristic attack not exposed in training. In addition, our newly designed ownership-based utility functions ensure that Flip-PSRO defenders maintain a high level of control while optimizing performance.