Anticipatory Reinforcement Learning for Trajectory Tracking
For practitioners applying DRL to industrial control, this work offers a low-overhead method to improve tracking accuracy, though the sim-to-real gap limits immediate practical impact.
The paper introduces a predictive state augmentation for DRL to reduce lag and overshoot in trajectory tracking, achieving a 9-fold error reduction in simulation (from 2.73° to 0.31° MAE) on a 1-DoF helicopter testbed, but with a sim-to-real gap. A simpler configuration with a single look-ahead horizon matched real-world top performance (1.11°).
Deep reinforcement learning (DRL) in industrial control often suffers from lag and overshoot due to purely reactive control based on the current tracking error. To achieve anticipatory control without high computational overhead, we introduce a predictive formulation that augments the DRL state space with target velocities and future reference horizons. Evaluating eight configurations using proximal policy optimization (PPO) on a 1-degree-of-freedom (1-DoF) helicopter testbed, simulation results showed a 9-fold error reduction, lowering the mean absolute deviation from 2.73° to 0.31°. However, zero-shot transfer to physical hardware revealed a sim-to-real gap. Interestingly, a simpler configuration using a single, further look-ahead horizon matched the real-world top performance of the most complex model (1.11°). Overall, evaluating various combinations of prediction horizons and target velocities demonstrated that highly granular predictive data is not necessarily required for physical transfer.