$q$-Munchausen Reinforcement Learning
This work addresses a specific technical bottleneck in reinforcement learning for researchers, offering an incremental improvement to M-RL by extending it to Tsallis entropy frameworks.
The paper tackled the performance degradation of Munchausen Reinforcement Learning (M-RL) when using Tsallis sparsemax policies, showing that a mismatch between conventional logarithms and Tsallis entropy caused flat learning curves. By introducing $q$-logarithm/exponential functions to align with Tsallis statistics, the proposed method achieved superior performance on benchmark problems, enabling more general M-RL with various entropic indices.
The recently successful Munchausen Reinforcement Learning (M-RL) features implicit Kullback-Leibler (KL) regularization by augmenting the reward function with logarithm of the current stochastic policy. Though significant improvement has been shown with the Boltzmann softmax policy, when the Tsallis sparsemax policy is considered, the augmentation leads to a flat learning curve for almost every problem considered. We show that it is due to the mismatch between the conventional logarithm and the non-logarithmic (generalized) nature of Tsallis entropy. Drawing inspiration from the Tsallis statistics literature, we propose to correct the mismatch of M-RL with the help of $q$-logarithm/exponential functions. The proposed formulation leads to implicit Tsallis KL regularization under the maximum Tsallis entropy framework. We show such formulation of M-RL again achieves superior performance on benchmark problems and sheds light on more general M-RL with various entropic indices $q$.