Distributional Reinforcement Learning with Regularized Wasserstein Loss
This work addresses the challenge of selecting appropriate distribution divergences in RL for researchers and practitioners, offering an incremental improvement over existing methods.
The paper tackled the problem of improving distributional reinforcement learning by proposing SinkhornDRL, which uses Sinkhorn divergence to minimize differences between Bellman return distributions, and showed that it consistently outperforms or matches existing algorithms on Atari games, with particular strength in multi-dimensional reward settings.
The empirical success of distributional reinforcement learning (RL) highly relies on the choice of distribution divergence equipped with an appropriate distribution representation. In this paper, we propose \textit{Sinkhorn distributional RL (SinkhornDRL)}, which leverages Sinkhorn divergence, a regularized Wasserstein loss, to minimize the difference between current and target Bellman return distributions. Theoretically, we prove the contraction properties of SinkhornDRL, aligning with the interpolation nature of Sinkhorn divergence between Wasserstein distance and Maximum Mean Discrepancy (MMD). The introduced SinkhornDRL enriches the family of distributional RL algorithms, contributing to interpreting the algorithm behaviors compared with existing approaches by our investigation into their relationships. Empirically, we show that SinkhornDRL consistently outperforms or matches existing algorithms on the Atari games suite and particularly stands out in the multi-dimensional reward setting. \thanks{Code is available in \url{https://github.com/datake/SinkhornDistRL}.}.