Convergence Theorems for Entropy-Regularized and Distributional Reinforcement Learning
This work addresses the interpretability and diversity issues in reinforcement learning policies, providing a theoretical foundation for more predictable and varied optimal policies, though it appears incremental as it builds on existing entropy-regularized methods.
The authors tackled the problem of reinforcement learning methods lacking interpretability and diversity in learned policies by introducing a theoretical framework with entropy regularization and temperature decoupling, which guarantees convergence to an interpretable, diversity-preserving optimal policy, such as one that samples all optimal actions uniformly, and enables accurate estimation of return distributions.
In the pursuit of finding an optimal policy, reinforcement learning (RL) methods generally ignore the properties of learned policies apart from their expected return. Thus, even when successful, it is difficult to characterize which policies will be learned and what they will do. In this work, we present a theoretical framework for policy optimization that guarantees convergence to a particular optimal policy, via vanishing entropy regularization and a temperature decoupling gambit. Our approach realizes an interpretable, diversity-preserving optimal policy as the regularization temperature vanishes and ensures the convergence of policy derived objects--value functions and return distributions. In a particular instance of our method, for example, the realized policy samples all optimal actions uniformly. Leveraging our temperature decoupling gambit, we present an algorithm that estimates, to arbitrary accuracy, the return distribution associated to its interpretable, diversity-preserving optimal policy.