Distributional Bellman Operators over Mean Embeddings
This work addresses distributional reinforcement learning for AI/ML researchers, presenting a novel framework that combines with deep RL to achieve better performance.
The paper tackles the problem of distributional reinforcement learning by proposing a framework using finite-dimensional mean embeddings of return distributions, deriving new algorithms with asymptotic convergence theory and demonstrating empirical improvements over baseline distributional approaches on the Arcade Learning Environment.
We propose a novel algorithmic framework for distributional reinforcement learning, based on learning finite-dimensional mean embeddings of return distributions. We derive several new algorithms for dynamic programming and temporal-difference learning based on this framework, provide asymptotic convergence theory, and examine the empirical performance of the algorithms on a suite of tabular tasks. Further, we show that this approach can be straightforwardly combined with deep reinforcement learning, and obtain a new deep RL agent that improves over baseline distributional approaches on the Arcade Learning Environment.