An Information-Geometric Approach to Artificial Curiosity
This work addresses the problem of designing intrinsic rewards for exploration in reinforcement learning, offering a foundational framework that integrates and constrains existing methods, though it is incremental in its theoretical unification.
The paper tackles the challenge of sparse rewards in reinforcement learning by proposing an information-geometric approach to formulate intrinsic rewards for artificial curiosity, uniquely constraining them to concave functions of reciprocal occupancy and linking them to existing methods like count-based and maximum entropy exploration.
Learning in environments with sparse rewards remains a fundamental challenge in reinforcement learning. Artificial curiosity addresses this limitation through intrinsic rewards to guide exploration, however, the precise formulation of these rewards has remained elusive. Ideally, such rewards should depend on the agent's information about the environment, remaining agnostic to the representation of the information -- an invariance central to information geometry. Leveraging information geometry, we show that invariance under congruent Markov morphisms and the agent-environment interaction, uniquely constrains intrinsic rewards to concave functions of the reciprocal occupancy. Additional geometrically motivated restrictions effectively limits the candidates to those determined by a real parameter that governs the occupancy space geometry. Remarkably, special values of this parameter are found to correspond to count-based and maximum entropy exploration, revealing a geometric exploration-exploitation trade-off. This framework provides important constraints to the engineering of intrinsic reward while integrating foundational exploration methods into a single, cohesive model.