Proximal Policy Optimization in Path Space: A Schrödinger Bridge Perspective
This work addresses a central challenge in on-policy reinforcement learning for generative policies, offering an incremental improvement by adapting PPO to trajectory-level processes.
The paper tackled the challenge of applying proximal policy optimization (PPO) to generative policies by proposing GSB-PPO, a path-space formulation inspired by the Generalized Schrödinger Bridge, which lifts updates from actions to trajectories; experimental results showed that the penalty-based objective consistently delivered better stability and performance than the clipping counterpart.
On-policy reinforcement learning with generative policies is promising but remains underexplored. A central challenge is that proximal policy optimization (PPO) is traditionally formulated in terms of action-space probability ratios, whereas diffusion- and flow-based policies are more naturally represented as trajectory-level generative processes. In this work, we propose GSB-PPO, a path-space formulation of generative PPO inspired by the Generalized Schrödinger Bridge (GSB). Our framework lifts PPO-style proximal updates from terminal actions to full generation trajectories, yielding a unified view of on-policy optimization for generative policies. Within this framework, we develop two concrete objectives: a clipping-based objective, GSB-PPO-Clip, and a penalty-based objective, GSB-PPO-Penalty. Experimental results show that while both objectives are compatible with on-policy training, the penalty formulation consistently delivers better stability and performance than the clipping counterpart. Overall, our results highlight path-space proximal regularization as an effective principle for training generative policies with PPO.