A Regularized Approach to Sparse Optimal Policy in Reinforcement Learning
This work addresses the need for multi-modal and sparse policies in reinforcement learning, offering a theoretical and practical framework that is incremental over existing entropy-regularized methods.
The paper tackles the problem of finding sparse optimal policies in reinforcement learning by proposing a general framework for regularized Markov decision processes, which includes entropy regularization as a special case and introduces conditions for inducing sparsity. The result includes a full mathematical analysis, generic methods for regularization, and empirical comparisons showing performance in discrete and continuous environments.
We propose and study a general framework for regularized Markov decision processes (MDPs) where the goal is to find an optimal policy that maximizes the expected discounted total reward plus a policy regularization term. The extant entropy-regularized MDPs can be cast into our framework. Moreover, under our framework, many regularization terms can bring multi-modality and sparsity, which are potentially useful in reinforcement learning. In particular, we present sufficient and necessary conditions that induce a sparse optimal policy. We also conduct a full mathematical analysis of the proposed regularized MDPs, including the optimality condition, performance error, and sparseness control. We provide a generic method to devise regularization forms and propose off-policy actor critic algorithms in complex environment settings. We empirically analyze the numerical properties of optimal policies and compare the performance of different sparse regularization forms in discrete and continuous environments.