Max-Entropy Reinforcement Learning with Flow Matching and A Case Study on LQR
This work addresses the trade-off between efficiency and expressiveness in reinforcement learning policies for researchers and practitioners, though it is incremental as it builds on existing SAC and flow-based methods.
The paper tackles the problem of limited expressiveness in max-entropy reinforcement learning by proposing a SAC variant that uses flow-based models for policy parameterization, and demonstrates that the algorithm learns the optimal action distribution in a linear quadratic regulator case study.
Soft actor-critic (SAC) is a popular algorithm for max-entropy reinforcement learning. In practice, the energy-based policies in SAC are often approximated using simple policy classes for efficiency, sacrificing the expressiveness and robustness. In this paper, we propose a variant of the SAC algorithm that parameterizes the policy with flow-based models, leveraging their rich expressiveness. In the algorithm, we evaluate the flow-based policy utilizing the instantaneous change-of-variable technique and update the policy with an online variant of flow matching developed in this paper. This online variant, termed importance sampling flow matching (ISFM), enables policy update with only samples from a user-specified sampling distribution rather than the unknown target distribution. We develop a theoretical analysis of ISFM, characterizing how different choices of sampling distributions affect the learning efficiency. Finally, we conduct a case study of our algorithm on the max-entropy linear quadratic regulator problems, demonstrating that the proposed algorithm learns the optimal action distribution.