Enhancing Control Policy Smoothness by Aligning Actions with Predictions from Preceding States
This addresses the challenge of applying deep reinforcement learning in real-world environments by reducing action oscillations, though it is an incremental improvement over prior loss-based methods.
The paper tackled the problem of high-frequency action oscillations in deep reinforcement learning for control tasks, proposing a loss-based method called ASAP that aligns actions with predictions from preceding states, resulting in smoother control and improved policy performance in experiments on Gymnasium and Isaac-Lab environments.
Deep reinforcement learning has proven to be a powerful approach to solving control tasks, but its characteristic high-frequency oscillations make it difficult to apply in real-world environments. While prior methods have addressed action oscillations via architectural or loss-based methods, the latter typically depend on heuristic or synthetic definitions of state similarity to promote action consistency, which often fail to accurately reflect the underlying system dynamics. In this paper, we propose a novel loss-based method by introducing a transition-induced similar state. The transition-induced similar state is defined as the distribution of next states transitioned from the previous state. Since it utilizes only environmental feedback and actually collected data, it better captures system dynamics. Building upon this foundation, we introduce Action Smoothing by Aligning Actions with Predictions from Preceding States (ASAP), an action smoothing method that effectively mitigates action oscillations. ASAP enforces action smoothness by aligning the actions with those taken in transition-induced similar states and by penalizing second-order differences to suppress high-frequency oscillations. Experiments in Gymnasium and Isaac-Lab environments demonstrate that ASAP yields smoother control and improved policy performance over existing methods.