PACER: A Fully Push-forward-based Distributional Reinforcement Learning Algorithm
This work addresses a bottleneck in distributional RL for researchers and practitioners by enabling exploration of a broader policy space beyond Gaussian assumptions, though it appears incremental as it builds on existing push-forward concepts.
The paper tackles the problem of limited policy classes in distributional reinforcement learning by proposing PACER, a fully push-forward-based algorithm that models return distributions and stochastic policies using push-forward operators, resulting in superior performance over state-of-the-art methods as validated by experiments.
In this paper, we propose the first fully push-forward-based distributional reinforcement learning algorithm, named PACER, which consists of a distributional critic, a stochastic actor and a sample-based encourager. Specifically, the push-forward operator is leveraged in both the critic and actor to model the return distributions and stochastic policies respectively, enabling them with equal modeling capability and thus enhancing the synergetic performance. Since it is infeasible to obtain the density function of the push-forward policies, novel sample-based regularizers are integrated in the encourager to incentivize efficient exploration and alleviate the risk of trapping into local optima. Moreover, a sample-based stochastic utility value policy gradient is established for the push-forward policy update, which circumvents the explicit demand of the policy density function in existing REINFORCE-based stochastic policy gradient. As a result, PACER fully utilizes the modeling capability of the push-forward operator and is able to explore a broader class of the policy space, compared with limited policy classes used in existing distributional actor critic algorithms (i.e. Gaussians). We validate the critical role of each component in our algorithm with extensive empirical studies. Experimental results demonstrate the superiority of our algorithm over the state-of-the-art.