Chat2Map: Efficient Scene Mapping from Multi-Ego Conversations
This addresses the challenge of scene mapping for robotics or AR/VR applications in a cost-efficient way, though it is incremental as it builds on existing audio-visual and mapping methods.
The paper tackles the problem of efficiently mapping a 3D scene from multiple egocentric conversational videos by proposing an audio-visual deep reinforcement learning approach that selectively samples visual data to reduce redundancy and power use, achieving a state-of-the-art cost-accuracy tradeoff in both simulation and real-world evaluations.
Can conversational videos captured from multiple egocentric viewpoints reveal the map of a scene in a cost-efficient way? We seek to answer this question by proposing a new problem: efficiently building the map of a previously unseen 3D environment by exploiting shared information in the egocentric audio-visual observations of participants in a natural conversation. Our hypothesis is that as multiple people ("egos") move in a scene and talk among themselves, they receive rich audio-visual cues that can help uncover the unseen areas of the scene. Given the high cost of continuously processing egocentric visual streams, we further explore how to actively coordinate the sampling of visual information, so as to minimize redundancy and reduce power use. To that end, we present an audio-visual deep reinforcement learning approach that works with our shared scene mapper to selectively turn on the camera to efficiently chart out the space. We evaluate the approach using a state-of-the-art audio-visual simulator for 3D scenes as well as real-world video. Our model outperforms previous state-of-the-art mapping methods, and achieves an excellent cost-accuracy tradeoff. Project: http://vision.cs.utexas.edu/projects/chat2map.