SHAPO: Sharpness-Aware Policy Optimization for Safe Exploration
For safety-critical RL applications, SHAPO provides a principled method to enhance safe exploration by leveraging epistemic uncertainty, achieving better Pareto frontiers of safety and performance.
SHAPO introduces a sharpness-aware policy update rule that uses perturbed parameters to bias RL agents toward conservative behavior in under-explored regions, consistently improving both safety and task performance over baselines across continuous-control tasks.
Safe exploration is a prerequisite for deploying reinforcement learning (RL) agents in safety-critical domains. In this paper, we approach safe exploration through the lens of epistemic uncertainty, where the actor's sensitivity to parameter perturbations serves as a practical proxy for regions of high uncertainty. We propose Sharpness-Aware Policy Optimization (SHAPO), a sharpness-aware policy update rule that evaluates gradients at perturbed parameters, making policy updates pessimistic with respect to the actor's epistemic uncertainty. Analytically we show that this adjustment implicitly reweighs policy gradients, amplifying the influence of rare unsafe actions while tempering contributions from already safe ones, thereby biasing learning toward conservative behavior in under-explored regions. Across several continuous-control tasks, our method consistently improves both safety and task performance over existing baselines, significantly expanding their Pareto frontiers.