Conjugated Discrete Distributions for Distributional Reinforcement Learning
This addresses a key limitation in reinforcement learning algorithms for stochastic environments, offering a robust solution for practitioners dealing with non-deterministic processes.
The paper tackles the problem of handling varied reward magnitudes in reinforcement learning by proposing a conjugated distributional operator that guarantees theoretical convergence for a large class of transformations, and demonstrates state-of-the-art performance on 55 Atari 2600 games with sticky-actions.
In this work we continue to build upon recent advances in reinforcement learning for finite Markov processes. A common approach among previous existing algorithms, both single-actor and distributed, is to either clip rewards or to apply a transformation method on Q-functions to handle a large variety of magnitudes in real discounted returns. We theoretically show that one of the most successful methods may not yield an optimal policy if we have a non-deterministic process. As a solution, we argue that distributional reinforcement learning lends itself to remedy this situation completely. By the introduction of a conjugated distributional operator we may handle a large class of transformations for real returns with guaranteed theoretical convergence. We propose an approximating single-actor algorithm based on this operator that trains agents directly on unaltered rewards using a proper distributional metric given by the Cramér distance. To evaluate its performance in a stochastic setting we train agents on a suite of 55 Atari 2600 games using sticky-actions and obtain state-of-the-art performance compared to other well-known algorithms in the Dopamine framework.