Dynamic Decision Frequency with Continuous Options
This work addresses the challenge of inefficient decision-making in reinforcement learning for physical systems, offering a method to adapt control frequencies dynamically, though it appears incremental as it builds on temporal-abstraction RL.
The authors tackled the problem of fixed decision frequencies in reinforcement learning by proposing the Continuous-Time Continuous-Options (CTCO) framework, which allows agents to choose variable-duration sub-policies for adaptive control, and demonstrated that its performance remains unaffected by environment interaction frequency while facilitating exploration in a robotic arm task.
In classic reinforcement learning algorithms, agents make decisions at discrete and fixed time intervals. The duration between decisions becomes a crucial hyperparameter, as setting it too short may increase the problem's difficulty by requiring the agent to make numerous decisions to achieve its goal while setting it too long can result in the agent losing control over the system. However, physical systems do not necessarily require a constant control frequency, and for learning agents, it is often preferable to operate with a low frequency when possible and a high frequency when necessary. We propose a framework called Continuous-Time Continuous-Options (CTCO), where the agent chooses options as sub-policies of variable durations. These options are time-continuous and can interact with the system at any desired frequency providing a smooth change of actions. We demonstrate the effectiveness of CTCO by comparing its performance to classical RL and temporal-abstraction RL methods on simulated continuous control tasks with various action-cycle times. We show that our algorithm's performance is not affected by the choice of environment interaction frequency. Furthermore, we demonstrate the efficacy of CTCO in facilitating exploration in a real-world visual reaching task for a 7 DOF robotic arm with sparse rewards.