Learning Compact Models for Planning with Exogenous Processes
This work addresses the challenge of planning efficiency in complex environments with exogenous processes, such as busy robotic settings, though it appears incremental as it builds on existing MDP and mutual information concepts.
The paper tackles the problem of approximate model minimization for MDPs with exogenous state variables by formalizing a mask-learning approach to select a subset of exogenous variables for efficient planning, and it validates the method in simulated robotic manipulation domains with many interacting agents.
We address the problem of approximate model minimization for MDPs in which the state is partitioned into endogenous and (much larger) exogenous components. An exogenous state variable is one whose dynamics are independent of the agent's actions. We formalize the mask-learning problem, in which the agent must choose a subset of exogenous state variables to reason about when planning; doing planning in such a reduced state space can often be significantly more efficient than planning in the full model. We then explore the various value functions at play within this setting, and describe conditions under which a policy for a reduced model will be optimal for the full MDP. The analysis leads us to a tractable approximate algorithm that draws upon the notion of mutual information among exogenous state variables. We validate our approach in simulated robotic manipulation domains where a robot is placed in a busy environment, in which there are many other agents also interacting with the objects. Visit http://tinyurl.com/chitnis-exogenous for a supplementary video.