Intrinsically-Motivated Humans and Agents in Open-World Exploration
This work addresses the gap in intrinsic motivation between humans and AI agents, offering insights for better agent design in open-world exploration, though it is incremental as it builds on existing objectives.
The study compared human and AI exploration in an open-world environment, finding that Entropy and Empowerment objectives correlate with human progress, suggesting they improve intrinsic reward design for agents, with Entropy aiding early exploration and Empowerment later stages.
What drives exploration? Understanding intrinsic motivation is a long-standing challenge in both cognitive science and artificial intelligence; numerous objectives have been proposed and used to train agents, yet there remains a gap between human and agent exploration. We directly compare adults, children, and AI agents in a complex open-ended environment, Crafter, and study how common intrinsic objectives: Entropy, Information Gain, and Empowerment, relate to their behavior. We find that only Entropy and Empowerment are consistently positively correlated with human exploration progress, indicating that these objectives may better inform intrinsic reward design for agents. Furthermore, across agents and humans we observe that Entropy initially increases rapidly, then plateaus, while Empowerment increases continuously, suggesting that state diversity may provide more signal in early exploration, while advanced exploration should prioritize control. Finally, we find preliminary evidence that private speech utterances, and particularly goal verbalizations, may aid exploration in children. Our data is available at https://github.com/alyd/humans_in_crafter_data.