Information-Theoretic Methods for Planning and Learning in Partially Observable Markov Decision Processes
This work addresses the challenge of designing efficient agents under intrinsic information constraints, which is incremental as it builds on existing POMDP frameworks with a focus on information-theoretic methods.
The dissertation tackles the problem of planning and learning for bounded agents with information-processing constraints in partially observable Markov decision processes, developing optimization tools and analyzing convergence challenges, with specific solutions for linear-Gaussian dynamics and extensions to unknown models.
Bounded agents are limited by intrinsic constraints on their ability to process information that is available in their sensors and memory and choose actions and memory updates. In this dissertation, we model these constraints as information-rate constraints on communication channels connecting these various internal components of the agent. We make four major contributions detailed below and many smaller contributions detailed in each section. First, we formulate the problem of optimizing the agent under both extrinsic and intrinsic constraints and develop the main tools for solving it. Second, we identify another reason for the challenging convergence properties of the optimization algorithm, which is the bifurcation structure of the update operator near phase transitions. Third, we study the special case of linear-Gaussian dynamics and quadratic cost (LQG), where the optimal solution has a particularly simple and solvable form. Fourth, we explore the learning task, where the model of the world dynamics is unknown and sample-based updates are used instead.