Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow
This work addresses the problem of inefficient training in MaxEnt RL for robotics and simulation tasks, offering an incremental improvement over existing actor-critic methods.
The paper tackles the challenge of integrating policy evaluation and improvement in Maximum-Entropy Reinforcement Learning for continuous action spaces by proposing a framework based on Energy-Based Normalizing Flows, which achieves superior performance on MuJoCo and high-dimensional robotic tasks compared to baselines.
Existing Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) methods for continuous action spaces are typically formulated based on actor-critic frameworks and optimized through alternating steps of policy evaluation and policy improvement. In the policy evaluation steps, the critic is updated to capture the soft Q-function. In the policy improvement steps, the actor is adjusted in accordance with the updated soft Q-function. In this paper, we introduce a new MaxEnt RL framework modeled using Energy-Based Normalizing Flows (EBFlow). This framework integrates the policy evaluation steps and the policy improvement steps, resulting in a single objective training process. Our method enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation. Moreover, this design supports the modeling of multi-modal action distributions while facilitating efficient action sampling. To evaluate the performance of our method, we conducted experiments on the MuJoCo benchmark suite and a number of high-dimensional robotic tasks simulated by Omniverse Isaac Gym. The evaluation results demonstrate that our method achieves superior performance compared to widely-adopted representative baselines.