Embodied Agents for Efficient Exploration and Smart Scene Description
This work addresses the need for human-friendly interpretability in robotic platforms for human-populated environments, though it appears incremental as it builds on existing methods for exploration and captioning.
The paper tackles the problem of enabling embodied agents to explore unseen indoor environments and generate natural language descriptions of scenes, proposing an approach that combines visual robotic exploration and image captioning to maximize semantic knowledge and avoid repetitions, with experiments in simulated and real-world environments demonstrating effective description of the robot's point of view.
The development of embodied agents that can communicate with humans in natural language has gained increasing interest over the last years, as it facilitates the diffusion of robotic platforms in human-populated environments. As a step towards this objective, in this work, we tackle a setting for visual navigation in which an autonomous agent needs to explore and map an unseen indoor environment while portraying interesting scenes with natural language descriptions. To this end, we propose and evaluate an approach that combines recent advances in visual robotic exploration and image captioning on images generated through agent-environment interaction. Our approach can generate smart scene descriptions that maximize semantic knowledge of the environment and avoid repetitions. Further, such descriptions offer user-understandable insights into the robot's representation of the environment by highlighting the prominent objects and the correlation between them as encountered during the exploration. To quantitatively assess the performance of the proposed approach, we also devise a specific score that takes into account both exploration and description skills. The experiments carried out on both photorealistic simulated environments and real-world ones demonstrate that our approach can effectively describe the robot's point of view during exploration, improving the human-friendly interpretability of its observations.