Bridging the Gap: Enabling Soft Actor Critic for High Performance Legged Locomotion
For researchers and engineers working on sim-to-real transfer for legged robots, this work enables the use of sample-efficient SAC for continuous adaptation on real hardware, which was previously dominated by PPO.
The paper identifies why Soft Actor-Critic (SAC) underperforms Proximal Policy Optimization (PPO) in massively parallel legged locomotion training and introduces modifications (policy initialization, timeout-aware critic targets, multi-step return estimation) that enable SAC to match PPO's performance across multiple robot platforms and tasks.
Proximal Policy Optimization (PPO) has become the de facto standard for training legged robots, thanks to its robustness and scalability in massively parallel simulation environments like IsaacLab. However, its on-policy nature makes it inherently sample-inefficient, preventing its use for continuous adaptation and fine-tuning on real hardware. Soft Actor-Critic (SAC), by contrast, is an off-policy algorithm that can reuse past experience, making it a natural candidate for sim-to-real transfer workflows where the same algorithm can be used both in simulation and for online learning on the real robot. Despite these advantages, SAC has consistently failed to match PPO's empirical performance in massively parallel training settings. This work identifies the root causes of this gap and introduces targeted modifications, covering policy initialization, timeout-aware critic targets, and multi-step return estimation, that enable SAC to train stably at scale. Evaluated across multiple legged robot platforms and diverse locomotion tasks, our approach closes the performance gap with PPO entirely.