Near-Minimax-Optimal Distributional Reinforcement Learning with a Generative Model
This provides a theoretical foundation for distributional RL algorithms, addressing a specific bottleneck in the field, though it is incremental in building on prior work.
The paper tackles the problem of approximating return distributions in model-based distributional reinforcement learning by proposing a new algorithm proven to be near-minimax-optimal with a generative model, resolving an open theoretical question.
We propose a new algorithm for model-based distributional reinforcement learning (RL), and prove that it is minimax-optimal for approximating return distributions with a generative model (up to logarithmic factors), resolving an open question of Zhang et al. (2023). Our analysis provides new theoretical results on categorical approaches to distributional RL, and also introduces a new distributional Bellman equation, the stochastic categorical CDF Bellman equation, which we expect to be of independent interest. We also provide an experimental study comparing several model-based distributional RL algorithms, with several takeaways for practitioners.