Reduced-Order Model-Guided Reinforcement Learning for Demonstration-Free Humanoid Locomotion
This addresses the challenge of demonstration-free locomotion for humanoid robots, offering a method that bridges reward-only and imitation-based approaches, though it is incremental in combining existing techniques.
The paper tackled the problem of training humanoid walking without motion capture data or complex reward shaping by introducing a two-stage reinforcement learning framework that uses a reduced-order model to guide a full-body policy, resulting in stable, symmetric gaits with substantially lower tracking error than a baseline at speeds of 1 and 4 meters per second.
We introduce Reduced-Order Model-Guided Reinforcement Learning (ROM-GRL), a two-stage reinforcement learning framework for humanoid walking that requires no motion capture data or elaborate reward shaping. In the first stage, a compact 4-DOF (four-degree-of-freedom) reduced-order model (ROM) is trained via Proximal Policy Optimization. This generates energy-efficient gait templates. In the second stage, those dynamically consistent trajectories guide a full-body policy trained with Soft Actor--Critic augmented by an adversarial discriminator, ensuring the student's five-dimensional gait feature distribution matches the ROM's demonstrations. Experiments at 1 meter-per-second and 4 meter-per-second show that ROM-GRL produces stable, symmetric gaits with substantially lower tracking error than a pure-reward baseline. By distilling lightweight ROM guidance into high-dimensional policies, ROM-GRL bridges the gap between reward-only and imitation-based locomotion methods, enabling versatile, naturalistic humanoid behaviors without any human demonstrations.