Does the Markov Decision Process Fit the Data: Testing for the Markov Property in Sequential Decision Making
This work addresses the need for non-parametric validation of the Markov property in reinforcement learning, which is crucial for identifying optimal policies in complex decision processes like high-order MDPs and POMDPs.
The authors tackled the problem of testing the Markov assumption in sequential decision making by proposing a Forward-Backward Learning procedure, which they applied to synthetic and real-world mobile health data to demonstrate its utility.
The Markov assumption (MA) is fundamental to the empirical validity of reinforcement learning. In this paper, we propose a novel Forward-Backward Learning procedure to test MA in sequential decision making. The proposed test does not assume any parametric form on the joint distribution of the observed data and plays an important role for identifying the optimal policy in high-order Markov decision processes and partially observable MDPs. We apply our test to both synthetic datasets and a real data example from mobile health studies to illustrate its usefulness.