Structural Estimation of Partially Observable Markov Decision Processes
This work addresses structural estimation challenges in POMDPs for domains like equipment replacement, offering incremental improvements in methodology and analysis.
The paper tackles the problem of estimating the primitives of Partially Observable Markov Decision Processes (POMDPs) from observable history, providing conditions for identifiability without state dynamics knowledge and a soft policy gradient algorithm with finite-time convergence analysis. It applies this methodology to optimal equipment replacement, demonstrating robustness with synthetic and real data and characterizing misspecification risks when ignoring partial observability.
In many practical settings control decisions must be made under partial/imperfect information about the evolution of a relevant state variable. Partially Observable Markov Decision Processes (POMDPs) is a relatively well-developed framework for modeling and analyzing such problems. In this paper we consider the structural estimation of the primitives of a POMDP model based upon the observable history of the process. We analyze the structural properties of POMDP model with random rewards and specify conditions under which the model is identifiable without knowledge of the state dynamics. We consider a soft policy gradient algorithm to compute a maximum likelihood estimator and provide a finite-time characterization of convergence to a stationary point. We illustrate the estimation methodology with an application to optimal equipment replacement. In this context, replacement decisions must be made under partial/imperfect information on the true state (i.e. condition of the equipment). We use synthetic and real data to highlight the robustness of the proposed methodology and characterize the potential for misspecification when partial state observability is ignored.