Depth Control of Model-Free AUVs via Reinforcement Learning
This addresses depth tracking for AUVs in marine applications, but it is incremental as it applies existing RL techniques to a specific control problem.
The paper tackles depth control for autonomous underwater vehicles (AUVs) with unknown dynamics by proposing a model-free reinforcement learning algorithm, achieving performance comparable to model-based controllers like LQI and NMPC in simulations and validating it on a seafloor dataset from the South China Sea.
In this paper, we consider depth control problems of an autonomous underwater vehicle (AUV) for tracking the desired depth trajectories. Due to the unknown dynamical model of the AUV, the problems cannot be solved by most of model-based controllers. To this purpose, we formulate the depth control problems of the AUV as continuous-state, continuous-action Markov decision processes (MDPs) under unknown transition probabilities. Based on deterministic policy gradient (DPG) and neural network approximation, we propose a model-free reinforcement learning (RL) algorithm that learns a state-feedback controller from sampled trajectories of the AUV. To improve the performance of the RL algorithm, we further propose a batch-learning scheme through replaying previous prioritized trajectories. We illustrate with simulations that our model-free method is even comparable to the model-based controllers as LQI and NMPC. Moreover, we validate the effectiveness of the proposed RL algorithm on a seafloor data set sampled from the South China Sea.