SonoWorld: From One Image to a 3D Audio-Visual Scene
This addresses the incomplete immersion in visual scene generation by adding sound, enabling applications like free-viewpoint audio-visual rendering and audio-visual spatial source separation.
The paper tackles the problem of generating a 3D audio-visual scene from a single image, introducing the Image2AVScene task and SonoWorld framework, which achieves effective results as confirmed by quantitative evaluations and a user study.
Tremendous progress in visual scene generation now turns a single image into an explorable 3D world, yet immersion remains incomplete without sound. We introduce Image2AVScene, the task of generating a 3D audio-visual scene from a single image, and present SonoWorld, the first framework to tackle this challenge. From one image, our pipeline outpaints a 360° panorama, lifts it into a navigable 3D scene, places language-guided sound anchors, and renders ambisonics for point, areal, and ambient sources, yielding spatial audio aligned with scene geometry and semantics. Quantitative evaluations on a newly curated real-world dataset and a controlled user study confirm the effectiveness of our approach. Beyond free-viewpoint audio-visual rendering, we also demonstrate applications to one-shot acoustic learning and audio-visual spatial source separation. Project website: https://humathe.github.io/sonoworld/