Flow Matching for Offline Reinforcement Learning with Discrete Actions
This work addresses a broader range of offline RL settings, including discrete and multi-agent problems, though it is incremental as it adapts existing flow matching techniques.
The paper tackles the limitation of generative policies in offline reinforcement learning to continuous action spaces by extending flow matching to discrete actions, achieving robust performance in high-dimensional control, multi-modal decision-making, and multi-objective scenarios.
Generative policies based on diffusion models and flow matching have shown strong promise for offline reinforcement learning (RL), but their applicability remains largely confined to continuous action spaces. To address a broader range of offline RL settings, we extend flow matching to a general framework that supports discrete action spaces with multiple objectives. Specifically, we replace continuous flows with continuous-time Markov chains, trained using a Q-weighted flow matching objective. We then extend our design to multi-agent settings, mitigating the exponential growth of joint action spaces via a factorized conditional path. We theoretically show that, under idealized conditions, optimizing this objective recovers the optimal policy. Extensive experiments further demonstrate that our method performs robustly in practical scenarios, including high-dimensional control, multi-modal decision-making, and dynamically changing preferences over multiple objectives. Our discrete framework can also be applied to continuous-control problems through action quantization, providing a flexible trade-off between representational complexity and performance.