Accelerating Visual-Policy Learning through Parallel Differentiable Simulation
This work addresses the bottleneck of slow training in visual reinforcement learning for robotics and control tasks, offering an incremental improvement through efficient integration with differentiable simulation.
The paper tackles the problem of computationally expensive visual policy learning by proposing an algorithm that decouples rendering from the computation graph, reducing overhead and stabilizing optimization. The result is a method that significantly reduces training time and outperforms baselines, achieving a 4× improvement in final return on humanoid locomotion and learning a running policy in 4 hours on a single GPU.
In this work, we propose a computationally efficient algorithm for visual policy learning that leverages differentiable simulation and first-order analytical policy gradients. Our approach decouple the rendering process from the computation graph, enabling seamless integration with existing differentiable simulation ecosystems without the need for specialized differentiable rendering software. This decoupling not only reduces computational and memory overhead but also effectively attenuates the policy gradient norm, leading to more stable and smoother optimization. We evaluate our method on standard visual control benchmarks using modern GPU-accelerated simulation. Experiments show that our approach significantly reduces wall-clock training time and consistently outperforms all baseline methods in terms of final returns. Notably, on complex tasks such as humanoid locomotion, our method achieves a $4\times$ improvement in final return, and successfully learns a humanoid running policy within 4 hours on a single GPU.