q-exponential family for policy optimization
This work addresses policy optimization for reinforcement learning practitioners by introducing a flexible policy family, though it is incremental as it builds on existing actor-critic algorithms with a new parametrization.
The paper tackles the problem of policy optimization in reinforcement learning by proposing the use of the q-exponential family for policy parametrization, finding that heavy-tailed policies, such as Student's t-distribution, are more effective and stable than Gaussian policies, with consistent improvements in offline benchmark problems.
Policy optimization methods benefit from a simple and tractable policy parametrization, usually the Gaussian for continuous action spaces. In this paper, we consider a broader policy family that remains tractable: the $q$-exponential family. This family of policies is flexible, allowing the specification of both heavy-tailed policies ($q>1$) and light-tailed policies ($q<1$). This paper examines the interplay between $q$-exponential policies for several actor-critic algorithms conducted on both online and offline problems. We find that heavy-tailed policies are more effective in general and can consistently improve on Gaussian. In particular, we find the Student's t-distribution to be more stable than the Gaussian across settings and that a heavy-tailed $q$-Gaussian for Tsallis Advantage Weighted Actor-Critic consistently performs well in offline benchmark problems. Our code is available at \url{https://github.com/lingweizhu/qexp}.