ISOPO: Proximal policy gradients without pi-old
This addresses the computational inefficiency of existing proximal policy methods for researchers and practitioners in reinforcement learning, though it appears incremental as it builds on prior methods.
The paper introduces Isometric Policy Optimization (ISOPO), a method that approximates the natural policy gradient in a single step to improve efficiency in reinforcement learning, achieving this with negligible computational overhead compared to vanilla REINFORCE.
This note introduces Isometric Policy Optimization (ISOPO), an efficient method to approximate the natural policy gradient in a single gradient step. In comparison, existing proximal policy methods such as GRPO or CISPO use multiple gradient steps with variants of importance ratio clipping to approximate a natural gradient step relative to a reference policy. In its simplest form, ISOPO normalizes the log-probability gradient of each sequence in the Fisher metric before contracting with the advantages. Another variant of ISOPO transforms the microbatch advantages based on the neural tangent kernel in each layer. ISOPO applies this transformation layer-wise in a single backward pass and can be implemented with negligible computational overhead compared to vanilla REINFORCE.