Deep Learning for Markov Chains: Lyapunov Functions, Poisson's Equation, and Stationary Distributions
This provides an automated tool for stability analysis in Markovian models, benefiting researchers and practitioners in fields like queueing theory, though it is incremental as it applies existing neural network methods to a new domain.
The paper tackled the problem of constructing Lyapunov functions for Markov chains, which traditionally requires significant analytical effort, by using deep learning to automate this process and also adapt it to solve Poisson's equation and estimate stationary distributions, demonstrating effectiveness in examples from queueing theory.
Lyapunov functions are fundamental to establishing the stability of Markovian models, yet their construction typically demands substantial creativity and analytical effort. In this paper, we show that deep learning can automate this process by training neural networks to satisfy integral equations derived from first-transition analysis. Beyond stability analysis, our approach can be adapted to solve Poisson's equation and estimate stationary distributions. While neural networks are inherently function approximators on compact domains, it turns out that our approach remains effective when applied to Markov chains on non-compact state spaces. We demonstrate the effectiveness of this methodology through several examples from queueing theory and beyond.