Actor-Critic for Linearly-Solvable Continuous MDP with Partially Known Dynamics
This addresses robotic applications where system dynamics are partially known, offering an incremental improvement over existing methods for L-MDPs.
The paper tackles reinforcement learning for continuous MDPs with partially known dynamics by proposing an actor-critic method that learns without a model of uncontrolled dynamics or transition noise, requiring only control dynamics knowledge. It shows improved learning and policy performance on synthetic and real-world traffic simulation tests.
In many robotic applications, some aspects of the system dynamics can be modeled accurately while others are difficult to obtain or model. We present a novel reinforcement learning (RL) method for continuous state and action spaces that learns with partial knowledge of the system and without active exploration. It solves linearly-solvable Markov decision processes (L-MDPs), which are well suited for continuous state and action spaces, based on an actor-critic architecture. Compared to previous RL methods for L-MDPs and path integral methods which are model based, the actor-critic learning does not need a model of the uncontrolled dynamics and, importantly, transition noise levels; however, it requires knowing the control dynamics for the problem. We evaluate our method on two synthetic test problems, and one real-world problem in simulation and using real traffic data. Our experiments demonstrate improved learning and policy performance.