Hitting time for Markov decision process
This work addresses a theoretical gap in analyzing MDPs, which is incremental as it builds on existing concepts like PageRank.
The paper tackles the problem of defining hitting time for Markov decision processes (MDPs) by constructing an associated Markov process whose stationary distribution matches the MDP's normalized occupancy measure, and defines the hitting time based on this process.
We define the hitting time for a Markov decision process (MDP). We do not use the hitting time of the Markov process induced by the MDP because the induced chain may not have a stationary distribution. Even it has a stationary distribution, the stationary distribution may not coincide with the (normalized) occupancy measure of the MDP. We observe a relationship between the MDP and the PageRank. Using this observation, we construct an MP whose stationary distribution coincides with the normalized occupancy measure of the MDP and we define the hitting time of the MDP as the hitting time of the associated MP.