Acoustic Field Video for Multimodal Scene Understanding
This addresses scene understanding tasks that are underconstrained with visual and audio alone, offering a practical enhancement for applications in robotics and XR, though it is incremental as it builds on existing multimodal models.
The paper tackles the problem of multimodal scene understanding by introducing acoustic field video as a new input representation, showing that incorporating spatial acoustic data improves a state-of-the-art vision-language model's accuracy from 38.3% to 67.4% on a 402-scene evaluation set.
We introduce and explore a new multimodal input representation for vision-language models: acoustic field video. Unlike conventional video (RGB with stereo/mono audio), our video stream provides a spatially grounded visualization of sound intensity across a scene, offering a new and powerful dimension of perceptual understanding. Our real-time pipeline uses low-cost beamforming microphone arrays that are already common in smart speakers and increasingly present in robotics and XR headsets, yet this sensing capability remains unutilized for scene understanding. To assess the value of spatial acoustic information, we constructed an evaluation set of 402 question-answer scenes, comparing a state-of-the-art VLM given conventional video with and without paired acoustic field video. Results show a clear and consistent improvement when incorporating spatial acoustic data; the VLM we test improves from 38.3% correct to 67.4%. Our findings highlight that many everyday scene understanding tasks remain underconstrained when relying solely on visual and audio input, and that acoustic field data provides a promising and practical direction for multimodal reasoning. A video demo is available at https://daehwakim.com/seeingsound