Efficient On-Policy Reinforcement Learning via Exploration of Sparse Parameter Space
This work addresses a bottleneck in on-policy RL for researchers and practitioners by offering an incremental improvement to enhance policy-gradient methods like PPO and TRPO.
The paper tackles the problem of inefficient exploration in on-policy reinforcement learning by revealing that higher-performing solutions often lie in unexplored regions near policy checkpoints, and introduces ExploRLer, a pluggable pipeline that systematically probes these neighborhoods, achieving significant improvements over baselines in complex continuous control environments without increasing gradient updates.
Policy-gradient methods such as Proximal Policy Optimization (PPO) are typically updated along a single stochastic gradient direction, leaving the rich local structure of the parameter space unexplored. Previous work has shown that the surrogate gradient is often poorly correlated with the true reward landscape. Building on this insight, we visualize the parameter space spanned by policy checkpoints within an iteration and reveal that higher performing solutions often lie in nearby unexplored regions. To exploit this opportunity, we introduce ExploRLer, a pluggable pipeline that seamlessly integrates with on-policy algorithms such as PPO and TRPO, systematically probing the unexplored neighborhoods of surrogate on-policy gradient updates. Without increasing the number of gradient updates, ExploRLer achieves significant improvements over baselines in complex continuous control environments. Our results demonstrate that iteration-level exploration provides a practical and effective way to strengthen on-policy reinforcement learning and offer a fresh perspective on the limitations of the surrogate objective.