Reinforcement Learning with Elastic Time Steps
This addresses inefficiencies in RL for real-time applications like robotics, but it is incremental as it builds on existing variable time step methods.
The paper tackled the problem of fixed control rates in reinforcement learning leading to inefficiencies, by proposing MOSEAC, an algorithm that uses elastic time steps to dynamically adjust control frequency. The result showed MOSEAC significantly outperformed other variable time step approaches in energy efficiency and task effectiveness in a 3D racing game.
Traditional Reinforcement Learning (RL) policies are typically implemented with fixed control rates, often disregarding the impact of control rate selection. This can lead to inefficiencies as the optimal control rate varies with task requirements. We propose the Multi-Objective Soft Elastic Actor-Critic (MOSEAC), an off-policy actor-critic algorithm that uses elastic time steps to dynamically adjust the control frequency. This approach minimizes computational resources by selecting the lowest viable frequency. We show that MOSEAC converges and produces stable policies at the theoretical level, and validate our findings in a real-time 3D racing game. MOSEAC significantly outperformed other variable time step approaches in terms of energy efficiency and task effectiveness. Additionally, MOSEAC demonstrated faster and more stable training, showcasing its potential for real-world RL applications in robotics.