Efficient Reinforcement Learning using Linear Koopman Dynamics for Nonlinear Robotic Systems
This work addresses control challenges for robotic systems like arms and quadrupeds, offering an incremental improvement in efficiency and performance over existing methods.
The paper tackles optimal closed-loop control for nonlinear robotic systems by developing a model-based reinforcement learning framework using linear Koopman dynamics, achieving improved sample efficiency over model-free RL and superior control performance compared to model-based RL baselines in simulations and real-world hardware tests.
This paper presents a model-based reinforcement learning (RL) framework for optimal closed-loop control of nonlinear robotic systems. The proposed approach learns linear lifted dynamics through Koopman operator theory and integrates the resulting model into an actor-critic architecture for policy optimization, where the policy represents a parameterized closed-loop controller. To reduce computational cost and mitigate model rollout errors, policy gradients are estimated using one-step predictions of the learned dynamics rather than multi-step propagation. This leads to an online mini-batch policy gradient framework that enables policy improvement from streamed interaction data. The proposed framework is evaluated on several simulated nonlinear control benchmarks and two real-world hardware platforms, including a Kinova Gen3 robotic arm and a Unitree Go1 quadruped. Experimental results demonstrate improved sample efficiency over model-free RL baselines, superior control performance relative to model-based RL baselines, and control performance comparable to classical model-based methods that rely on exact system dynamics.