Non-Markovian policies occupancy measures
This provides a theoretical simplification for researchers and practitioners in reinforcement learning, though it is incremental as it builds on existing policy analysis frameworks.
The paper tackles the problem of analyzing non-Markovian policies in reinforcement learning by proving that their occupancy measures can be equivalently generated by Markovian policies, enabling direct extension of theorems and simplifying proofs, particularly for applications like replay buffers and datasets.
A central object of study in Reinforcement Learning (RL) is the Markovian policy, in which an agent's actions are chosen from a memoryless probability distribution, conditioned only on its current state. The family of Markovian policies is broad enough to be interesting, yet simple enough to be amenable to analysis. However, RL often involves more complex policies: ensembles of policies, policies over options, policies updated online, etc. Our main contribution is to prove that the occupancy measure of any non-Markovian policy, i.e., the distribution of transition samples collected with it, can be equivalently generated by a Markovian policy. This result allows theorems about the Markovian policy class to be directly extended to its non-Markovian counterpart, greatly simplifying proofs, in particular those involving replay buffers and datasets. We provide various examples of such applications to the field of Reinforcement Learning.