When to Sense and Control? A Time-adaptive Approach for Continuous-Time RL
This addresses the problem of manual intervention costs in applications like greenhouse control or medical treatments by enabling fewer interactions, though it is incremental as it builds on existing RL methods.
The paper tackles the challenge of costly interactions in continuous-time systems by introducing a time-adaptive RL framework (TaCoS) that optimizes policies to predict control durations, reducing interactions while maintaining or improving performance, with state-of-the-art RL algorithms showing drastic reductions and a model-based variant (OTaCoS) achieving sublinear regret and sample-efficiency gains.
Reinforcement learning (RL) excels in optimizing policies for discrete-time Markov decision processes (MDP). However, various systems are inherently continuous in time, making discrete-time MDPs an inexact modeling choice. In many applications, such as greenhouse control or medical treatments, each interaction (measurement or switching of action) involves manual intervention and thus is inherently costly. Therefore, we generally prefer a time-adaptive approach with fewer interactions with the system. In this work, we formalize an RL framework, Time-adaptive Control & Sensing (TaCoS), that tackles this challenge by optimizing over policies that besides control predict the duration of its application. Our formulation results in an extended MDP that any standard RL algorithm can solve. We demonstrate that state-of-the-art RL algorithms trained on TaCoS drastically reduce the interaction amount over their discrete-time counterpart while retaining the same or improved performance, and exhibiting robustness over discretization frequency. Finally, we propose OTaCoS, an efficient model-based algorithm for our setting. We show that OTaCoS enjoys sublinear regret for systems with sufficiently smooth dynamics and empirically results in further sample-efficiency gains.