Generalized Fitted Q-Iteration with Clustered Data
This addresses the problem of clustered data in reinforcement learning, particularly for healthcare applications, representing an incremental improvement with specific gains.
The paper tackles reinforcement learning with clustered data by proposing a generalized fitted Q-iteration algorithm that incorporates generalized estimating equations to handle intra-cluster correlations, achieving on average a half reduction in regret compared to standard FQI in simulations and a mobile health dataset.
This paper focuses on reinforcement learning (RL) with clustered data, which is commonly encountered in healthcare applications. We propose a generalized fitted Q-iteration (FQI) algorithm that incorporates generalized estimating equations into policy learning to handle the intra-cluster correlations. Theoretically, we demonstrate (i) the optimalities of our Q-function and policy estimators when the correlation structure is correctly specified, and (ii) their consistencies when the structure is mis-specified. Empirically, through simulations and analyses of a mobile health dataset, we find the proposed generalized FQI achieves, on average, a half reduction in regret compared to the standard FQI.