Markov Chain Concentration with an Application in Reinforcement Learning
This work provides incremental theoretical tools for analyzing concentration in Markov chains, with potential applications in reinforcement learning algorithms.
The paper tackles the problem of establishing subgaussian concentration for Lipschitz functions of random variables, deriving a simplified variance expression under Markov chain assumptions and weighted Hamming metrics, and applies these results to reinforcement learning.
Given $X_1,\cdot ,X_N$ random variables whose joint distribution is given as $μ$ we will use the Martingale Method to show any Lipshitz Function $f$ over these random variables is subgaussian. The Variance parameter however can have a simple expression under certain conditions. For example under the assumption that the random variables follow a Markov Chain and that the function is Lipschitz under a Weighted Hamming Metric. We shall conclude with certain well known techniques from concentration of suprema of random processes with applications in Reinforcement Learning