Learn A Flexible Exploration Model for Parameterized Action Markov Decision Processes
This work addresses a specific bottleneck in hybrid action models for RL, offering incremental improvements in efficiency and performance for complex environments.
The paper tackles the problem of low learning efficiency and information loss in Parameterized Action Markov Decision Processes (PAMDPs) for reinforcement learning, proposing FLEXplore, a model-based RL algorithm that improves learning efficiency and asymptotic performance, as demonstrated by empirical results on standard benchmarks.
Hybrid action models are widely considered an effective approach to reinforcement learning (RL) modeling. The current mainstream method is to train agents under Parameterized Action Markov Decision Processes (PAMDPs), which performs well in specific environments. Unfortunately, these models either exhibit drastic low learning efficiency in complex PAMDPs or lose crucial information in the conversion between raw space and latent space. To enhance the learning efficiency and asymptotic performance of the agent, we propose a model-based RL (MBRL) algorithm, FLEXplore. FLEXplore learns a parameterized-action-conditioned dynamics model and employs a modified Model Predictive Path Integral control. Unlike conventional MBRL algorithms, we carefully design the dynamics loss function and reward smoothing process to learn a loose yet flexible model. Additionally, we use the variational lower bound to maximize the mutual information between the state and the hybrid action, enhancing the exploration effectiveness of the agent. We theoretically demonstrate that FLEXplore can reduce the regret of the rollout trajectory through the Wasserstein Metric under given Lipschitz conditions. Our empirical results on several standard benchmarks show that FLEXplore has outstanding learning efficiency and asymptotic performance compared to other baselines.