Dyna-Style Reinforcement Learning Modeling and Control of Non-linear Dynamics
This work addresses the challenge of sample inefficiency in controlling complex nonlinear systems, such as robotics, by combining data-driven modeling with reinforcement learning, though it is incremental as it builds on existing methods like SINDy and TD3.
The paper tackled controlling nonlinear dynamical systems by proposing a Dyna-Style Reinforcement Learning framework that integrates SINDy for data-driven modeling with TD3 for policy learning, achieving superior accuracy and robustness in stabilization and trajectory tracking for a bi-rotor system compared to direct reinforcement learning methods.
Controlling systems with complex, nonlinear dynamics poses a significant challenge, particularly in achieving efficient and robust control. In this paper, we propose a Dyna-Style Reinforcement Learning control framework that integrates Sparse Identification of Nonlinear Dynamics (SINDy) with Twin Delayed Deep Deterministic Policy Gradient (TD3) reinforcement learning. SINDy is used to identify a data-driven model of the system, capturing its key dynamics without requiring an explicit physical model. This identified model is used to generate synthetic rollouts that are periodically injected into the reinforcement learning replay buffer during training on the real environment, enabling efficient policy learning with limited data available. By leveraging this hybrid approach, we mitigate the sample inefficiency of traditional model-free reinforcement learning methods while ensuring accurate control of nonlinear systems. To demonstrate the effectiveness of this framework, we apply it to a bi-rotor system as a case study, evaluating its performance in stabilization and trajectory tracking. The results show that our SINDy-TD3 approach achieves superior accuracy and robustness compared to direct reinforcement learning techniques, highlighting the potential of combining data-driven modeling with reinforcement learning for complex dynamical systems.