FastDSAC: Unlocking the Potential of Maximum Entropy RL in High-Dimensional Humanoid Control
This work addresses exploration inefficiency and training instability in high-dimensional continuous control for robotics, representing a strong specific gain rather than a broad paradigm shift.
The paper tackled the challenge of scaling Maximum Entropy Reinforcement Learning to high-dimensional humanoid control by introducing FastDSAC, which achieved notable gains of 180% and 400% on specific tasks like Basketball and Balance Hard.
Scaling Maximum Entropy Reinforcement Learning (RL) to high-dimensional humanoid control remains a formidable challenge, as the ``curse of dimensionality'' induces severe exploration inefficiency and training instability in expansive action spaces. Consequently, recent high-throughput paradigms have largely converged on deterministic policy gradients combined with massive parallel simulation. We challenge this compromise with FastDSAC, a framework that effectively unlocks the potential of maximum entropy stochastic policies for complex continuous control. We introduce Dimension-wise Entropy Modulation (DEM) to dynamically redistribute the exploration budget and enforce diversity, alongside a continuous distributional critic tailored to ensure value fidelity and mitigate high-dimensional value overestimation. Extensive evaluations on HumanoidBench and other continuous control tasks demonstrate that rigorously designed stochastic policies can consistently match or outperform deterministic baselines, achieving notable gains of 180\% and 400\% on the challenging \textit{Basketball} and \textit{Balance Hard} tasks.