Robust Reinforcement Learning under Diffusion Models for Data with Jumps
This work addresses a domain-specific problem for researchers and practitioners in continuous-time reinforcement learning, offering an incremental improvement by enhancing robustness against jumps in state dynamics.
The paper tackles the challenge of reinforcement learning in continuous-time settings with stochastic differential equations involving jumps, by introducing the Mean-Square Bipower Variation Error (MSBVE) algorithm, which reliably estimates the value function and surpasses the Mean-Square TD Error algorithm's performance in such environments.
Reinforcement Learning (RL) has proven effective in solving complex decision-making tasks across various domains, but challenges remain in continuous-time settings, particularly when state dynamics are governed by stochastic differential equations (SDEs) with jump components. In this paper, we address this challenge by introducing the Mean-Square Bipower Variation Error (MSBVE) algorithm, which enhances robustness and convergence in scenarios involving significant stochastic noise and jumps. We first revisit the Mean-Square TD Error (MSTDE) algorithm, commonly used in continuous-time RL, and highlight its limitations in handling jumps in state dynamics. The proposed MSBVE algorithm minimizes the mean-square quadratic variation error, offering improved performance over MSTDE in environments characterized by SDEs with jumps. Simulations and formal proofs demonstrate that the MSBVE algorithm reliably estimates the value function in complex settings, surpassing MSTDE's performance when faced with jump processes. These findings underscore the importance of alternative error metrics to improve the resilience and effectiveness of RL algorithms in continuous-time frameworks.