Quantile-Based Deep Reinforcement Learning using Two-Timescale Policy Gradient Algorithms
This work addresses a specific problem in reinforcement learning for scenarios requiring risk-aware decision-making, representing an incremental advancement.
The paper tackled the problem of optimizing quantiles of cumulative reward in reinforcement learning, proposing Quantile-Based Policy Optimization (QPO) and its variant QPPO, which outperformed existing baseline algorithms under the quantile criterion.
Classical reinforcement learning (RL) aims to optimize the expected cumulative reward. In this work, we consider the RL setting where the goal is to optimize the quantile of the cumulative reward. We parameterize the policy controlling actions by neural networks, and propose a novel policy gradient algorithm called Quantile-Based Policy Optimization (QPO) and its variant Quantile-Based Proximal Policy Optimization (QPPO) for solving deep RL problems with quantile objectives. QPO uses two coupled iterations running at different timescales for simultaneously updating quantiles and policy parameters, whereas QPPO is an off-policy version of QPO that allows multiple updates of parameters during one simulation episode, leading to improved algorithm efficiency. Our numerical results indicate that the proposed algorithms outperform the existing baseline algorithms under the quantile criterion.