Batch Stationary Distribution Estimation
This addresses the problem of stationary distribution estimation for researchers and practitioners in fields like statistics and machine learning, where only pre-collected data is available, representing a novel method for a known bottleneck rather than an incremental improvement.
The paper tackles the problem of estimating the stationary distribution of an ergodic Markov chain from a fixed set of sampled transitions, without additional access to the underlying process, by proposing a consistent estimator based on a correction ratio function. The result is a variational power method (VPM) that provides provably consistent estimates and yields significantly better estimates across various problems, such as queueing and off-policy evaluation.
We consider the problem of approximating the stationary distribution of an ergodic Markov chain given a set of sampled transitions. Classical simulation-based approaches assume access to the underlying process so that trajectories of sufficient length can be gathered to approximate stationary sampling. Instead, we consider an alternative setting where a fixed set of transitions has been collected beforehand, by a separate, possibly unknown procedure. The goal is still to estimate properties of the stationary distribution, but without additional access to the underlying system. We propose a consistent estimator that is based on recovering a correction ratio function over the given data. In particular, we develop a variational power method (VPM) that provides provably consistent estimates under general conditions. In addition to unifying a number of existing approaches from different subfields, we also find that VPM yields significantly better estimates across a range of problems, including queueing, stochastic differential equations, post-processing MCMC, and off-policy evaluation.