Instance-Dependent Continuous-Time Reinforcement Learning via Maximum Likelihood Estimation
This work addresses the challenge of instance-dependent behavior in CTRL for dynamic environments, offering a novel approach that is incremental in improving sample efficiency and adaptability.
The paper tackled the problem of continuous-time reinforcement learning (CTRL) adapting to varying difficulty levels by introducing a model-based algorithm using maximum likelihood estimation to estimate state marginal density, resulting in a regret bound scaling with reward variance and measurement resolution that becomes independent of measurement strategy with adaptive observation frequency.
Continuous-time reinforcement learning (CTRL) provides a natural framework for sequential decision-making in dynamic environments where interactions evolve continuously over time. While CTRL has shown growing empirical success, its ability to adapt to varying levels of problem difficulty remains poorly understood. In this work, we investigate the instance-dependent behavior of CTRL and introduce a simple, model-based algorithm built on maximum likelihood estimation (MLE) with a general function approximator. Unlike existing approaches that estimate system dynamics directly, our method estimates the state marginal density to guide learning. We establish instance-dependent performance guarantees by deriving a regret bound that scales with the total reward variance and measurement resolution. Notably, the regret becomes independent of the specific measurement strategy when the observation frequency adapts appropriately to the problem's complexity. To further improve performance, our algorithm incorporates a randomized measurement schedule that enhances sample efficiency without increasing measurement cost. These results highlight a new direction for designing CTRL algorithms that automatically adjust their learning behavior based on the underlying difficulty of the environment.