SAINT: Attention-Based Modeling of Sub-Action Dependencies in Multi-Action Policies
This addresses a key bottleneck in reinforcement learning for real-world applications with complex action spaces, though it appears incremental as an architectural improvement over existing methods.
The paper tackles the problem of combinatorial action spaces in reinforcement learning by introducing SAINT, a Transformer-based policy architecture that models sub-action dependencies via self-attention. In experiments across 15 environments with up to 17 million joint actions, SAINT consistently outperformed strong baselines.
The combinatorial structure of many real-world action spaces leads to exponential growth in the number of possible actions, limiting the effectiveness of conventional reinforcement learning algorithms. Recent approaches for combinatorial action spaces impose factorized or sequential structures over sub-actions, failing to capture complex joint behavior. We introduce the Sub-Action Interaction Network using Transformers (SAINT), a novel policy architecture that represents multi-component actions as unordered sets and models their dependencies via self-attention conditioned on the global state. SAINT is permutation-invariant, sample-efficient, and compatible with standard policy optimization algorithms. In 15 distinct combinatorial environments across three task domains, including environments with nearly 17 million joint actions, SAINT consistently outperforms strong baselines.