Reinforcement Learning in POMDP's via Direct Gradient Ascent
This provides a method for reinforcement learning in POMDPs, addressing challenges in environments with partial observability, but it is incremental as it builds on existing REINFORCE-like approaches.
The paper tackles the problem of optimizing policy performance in partially observable Markov decision processes (POMDPs) by introducing GPOMDP, a gradient-based algorithm that estimates the gradient of average reward using a single sample path and one free parameter, and proves its convergence for finding local optima.
This paper discusses theoretical and experimental aspects of gradient-based approaches to the direct optimization of policy performance in controlled POMDPs. We introduce GPOMDP, a REINFORCE-like algorithm for estimating an approximation to the gradient of the average reward as a function of the parameters of a stochastic policy. The algorithm's chief advantages are that it requires only a single sample path of the underlying Markov chain, it uses only one free parameter $β\in [0,1)$, which has a natural interpretation in terms of bias-variance trade-off, and it requires no knowledge of the underlying state. We prove convergence of GPOMDP and show how the gradient estimates produced by GPOMDP can be used in a conjugate-gradient procedure to find local optima of the average reward.