Ask & Explore: Grounded Question Answering for Curiosity-Driven Exploration
This work addresses the challenge of exploration in sparse-reward scenarios for AI agents, offering a novel approach that is incremental over existing curiosity methods.
The paper tackles the problem of sparse extrinsic rewards in reinforcement learning by proposing a curiosity-driven exploration method based on grounded question answering, which leads to more efficient task solving by uncovering specific environmental knowledge.
In many real-world scenarios where extrinsic rewards to the agent are extremely sparse, curiosity has emerged as a useful concept providing intrinsic rewards that enable the agent to explore its environment and acquire information to achieve its goals. Despite their strong performance on many sparse-reward tasks, existing curiosity approaches rely on an overly holistic view of state transitions, and do not allow for a structured understanding of specific aspects of the environment. In this paper, we formulate curiosity based on grounded question answering by encouraging the agent to ask questions about the environment and be curious when the answers to these questions change. We show that natural language questions encourage the agent to uncover specific knowledge about their environment such as the physical properties of objects as well as their spatial relationships with other objects, which serve as valuable curiosity rewards to solve sparse-reward tasks more efficiently.