Learn Quasi-stationary Distributions of Finite State Markov Chain
This provides a new computational method for a specific mathematical problem in Markov chain analysis, though it appears incremental as an application of existing RL techniques.
The authors tackled the problem of computing quasi-stationary distributions for finite state Markov chains by proposing a reinforcement learning approach that minimizes KL-divergence between path distributions, and demonstrated it with numerical examples.
We propose a reinforcement learning (RL) approach to compute the expression of quasi-stationary distribution. Based on the fixed-point formulation of quasi-stationary distribution, we minimize the KL-divergence of two Markovian path distributions induced by the candidate distribution and the true target distribution. To solve this challenging minimization problem by gradient descent, we apply the reinforcement learning technique by introducing the reward and value functions. We derive the corresponding policy gradient theorem and design an actor-critic algorithm to learn the optimal solution and the value function. The numerical examples of finite state Markov chain are tested to demonstrate the new method.