From Curiosity to Competence: How World Models Interact with the Dynamics of Exploration
This work addresses the problem of adaptive exploration in reinforcement learning, offering insights for cognitive theories and efficient learning, though it is incremental in bridging existing theories with RL methods.
The paper investigates how agents balance curiosity and competence in exploration, comparing tabular and world model-based agents, finding that prioritizing both improves exploration and revealing interactions between exploration and representation learning.
What drives an agent to explore the world while also maintaining control over the environment? From a child at play to scientists in the lab, intelligent agents must balance curiosity (the drive to seek knowledge) with competence (the drive to master and control the environment). Bridging cognitive theories of intrinsic motivation with reinforcement learning, we ask how evolving internal representations mediate the trade-off between curiosity (novelty or information gain) and competence (empowerment). We compare two model-based agents using handcrafted state abstractions (Tabular) or learning an internal world model (Dreamer). The Tabular agent shows curiosity and competence guide exploration in distinct patterns, while prioritizing both improves exploration. The Dreamer agent reveals a two-way interaction between exploration and representation learning, mirroring the developmental co-evolution of curiosity and competence. Our findings formalize adaptive exploration as a balance between pursuing the unknown and the controllable, offering insights for cognitive theories and efficient reinforcement learning.