Markov Decision Processes with Noisy State Observation
This work addresses a common issue in reinforcement learning for real-world applications, offering incremental improvements in state estimation techniques for noisy environments.
The paper tackles the problem of noisy state observations in Markov Decision Processes by modeling uncertainty with a confusion matrix and proposing two novel algorithmic approaches for noise estimation, demonstrating their effectiveness through simulations in various scenarios.
This paper addresses the challenge of a particular class of noisy state observations in Markov Decision Processes (MDPs), a common issue in various real-world applications. We focus on modeling this uncertainty through a confusion matrix that captures the probabilities of misidentifying the true state. Our primary goal is to estimate the inherent measurement noise, and to this end, we propose two novel algorithmic approaches. The first, the method of second-order repetitive actions, is designed for efficient noise estimation within a finite time window, providing identifiable conditions for system analysis. The second approach comprises a family of Bayesian algorithms, which we thoroughly analyze and compare in terms of performance and limitations. We substantiate our theoretical findings with simulations, demonstrating the effectiveness of our methods in different scenarios, particularly highlighting their behavior in environments with varying stationary distributions. Our work advances the understanding of reinforcement learning in noisy environments, offering robust techniques for more accurate state estimation in MDPs.