Local Guidance, Global Impact: Gaussian-Reshaped Trust Region Unlocks Behavior Transitions
For reinforcement learning practitioners dealing with non-stationary environments, GTR offers a simple, architecture-agnostic improvement over PPO that enables robust behavior transitions.
Proximal Policy Optimization (PPO) fails in non-stationary environments due to directionally inefficient updates and divergence-based regularization that discourages necessary large policy shifts. The proposed Gaussian Trust Region Policy Optimization (GTR) uses a Gaussian kernel to reshape the trust region, providing bounded, non-monotonic constraints that enable effective adaptation, achieving strong performance across games, robotics, open-world exploration, and language model post-training.
While Proximal Policy Optimization (PPO) demonstrates strong performance in stationary settings, we show that its standard optimization paradigm struggles in continual and non-stationary environments. The failure does not stem from insufficient model capacity or overly restrictive clipping. Instead, PPO performs persistent, directionally inefficient local updates, which indicates a lack of geometry-aware guidance for accumulating meaningful behavioral change and ultimately hindering transitions toward new behavior patterns. Although divergence-based regularization introduces partial geometric awareness, its monotonically increasing penalties implicitly discourage large policy deviations, even when such shifts are necessary for effective adaptation. To address this limitation, we propose Gaussian Trust Region Policy Optimization (GTR), which reshapes the trust region using a Gaussian kernel. The resulting constraint is bounded and non-monotonic, providing strong local stability while progressively relaxing under sustained high-advantage updates. To further improve robustness, we introduce a Mixture Gaussian Anchor that adapts to recent policy trajectories, reducing variance induced by stale references. GTR is architecture-agnostic and achieves strong performance across games, simulated robotic control, open-world exploration, and language model post-training. These results demonstrate that geometry-aware trust-region design can be a promising direction for robust reinforcement learning in complex non-stationary environments. Our code is available at https://anonymous.4open.science/r/GTR_demo/README.md.