Information is Power: Intrinsic Control via Information Capture
This addresses intrinsic motivation for AI agents in partially-observed settings, offering a novel objective but with incremental improvements over existing methods.
The paper tackles the problem of defining a general-purpose objective for artificial agents to acquire skills in partially-observed environments without extrinsic rewards, proposing to minimize entropy of state visitation via a latent model, and demonstrates that the agent learns to control dynamic objects in visual environments.
Humans and animals explore their environment and acquire useful skills even in the absence of clear goals, exhibiting intrinsic motivation. The study of intrinsic motivation in artificial agents is concerned with the following question: what is a good general-purpose objective for an agent? We study this question in dynamic partially-observed environments, and argue that a compact and general learning objective is to minimize the entropy of the agent's state visitation estimated using a latent state-space model. This objective induces an agent to both gather information about its environment, corresponding to reducing uncertainty, and to gain control over its environment, corresponding to reducing the unpredictability of future world states. We instantiate this approach as a deep reinforcement learning agent equipped with a deep variational Bayes filter. We find that our agent learns to discover, represent, and exercise control of dynamic objects in a variety of partially-observed environments sensed with visual observations without extrinsic reward.