RN-D: Discretized Categorical Actors with Regularized Networks for On-Policy Reinforcement Learning
This addresses optimization stability issues in continuous control RL, though it appears incremental as it builds on existing architectural advances.
The paper tackled the problem of brittle optimization in on-policy deep reinforcement learning for continuous control by replacing standard Gaussian actors with discretized categorical actors and regularized networks, achieving state-of-the-art performance across diverse benchmarks.
On-policy deep reinforcement learning remains a dominant paradigm for continuous control, yet standard implementations rely on Gaussian actors and relatively shallow MLP policies, often leading to brittle optimization when gradients are noisy and policy updates must be conservative. In this paper, we revisit policy representation as a first-class design choice for on-policy optimization. We study discretized categorical actors that represent each action dimension with a distribution over bins, yielding a policy objective that resembles a cross-entropy loss. Building on architectural advances from supervised learning, we further propose regularized actor networks, while keeping critic design fixed. Our results show that simply replacing the standard actor network with our discretized regularized actor yields consistent gains and achieve the state-of-the-art performance across diverse continuous-control benchmarks.