D2C-HRHR: Discrete Actions with Double Distributional Critics for High-Risk-High-Return Tasks
This addresses the challenge of handling multimodal action distributions and stochastic returns in RL for domains like robotics, though it is an incremental improvement over prior methods.
The paper tackled the problem of reinforcement learning in high-risk-high-return tasks, where existing methods with unimodal policies are ineffective, by proposing a framework that discretizes actions and uses dual critics, resulting in outperformance on locomotion and manipulation benchmarks.
Tasks involving high-risk-high-return (HRHR) actions, such as obstacle crossing, often exhibit multimodal action distributions and stochastic returns. Most reinforcement learning (RL) methods assume unimodal Gaussian policies and rely on scalar-valued critics, which limits their effectiveness in HRHR settings. We formally define HRHR tasks and theoretically show that Gaussian policies cannot guarantee convergence to the optimal solution. To address this, we propose a reinforcement learning framework that (i) discretizes continuous action spaces to approximate multimodal distributions, (ii) employs entropy-regularized exploration to improve coverage of risky but rewarding actions, and (iii) introduces a dual-critic architecture for more accurate discrete value distribution estimation. The framework scales to high-dimensional action spaces, supporting complex control domains. Experiments on locomotion and manipulation benchmarks with high risks of failure demonstrate that our method outperforms baselines, underscoring the importance of explicitly modeling multimodality and risk in RL.