Steering Generative Reinforcement Learning into Stable Robotic Controller
For roboticists using generative reinforcement learning, SteerGenPO addresses the instability of diffusion policies, enabling precise and robust control in high-dimensional systems.
SteerGenPO replaces stochastic latent sampling in diffusion-based generative policies with a learned deterministic latent actor, improving stability and control in high-dimensional robotic systems. It achieves superior performance over classical and generative RL baselines on six Isaac Lab benchmarks and a Unitree G1 locomotion task.
Diffusion and flow-based generative policies provide a powerful policy class for reinforcement learning by inducing rich stochastic exploration through iterative action generation. However, the stochasticity of diffusion policies is not suitable for stable and precise control in high-dimensional robotic systems, where small action variations can accumulate into inconsistent motion and reduced robustness. To address this issue, we propose SteerGenPO, a latent-space reinforcement learning framework that steers a trained generative policy into a robust deterministic robotic controller. The key idea is to replace stochastic latent sampling of the trained generative policy with a learned latent actor that predicts a state-dependent latent input for the generative policies. This separates exploration and control: stochastic generative sampling provides diverse action proposals during policy learning, while deterministic latent steering provides stable and adaptive control at deployment. We evaluate SteerGenPO on six Isaac Lab benchmarks and a Unitree G1 locomotion task. The results show SteerGenPO improves over both classical RL and generative RL baselines, while its deterministic latent steering produces more stable inference-time behaviors and more reliable command responses.