How Would It Sound? Material-Controlled Multimodal Acoustic Profile Generation for Indoor Scenes
This work addresses the need for customizable acoustic simulation in indoor environments, offering a tool for designers and engineers, though it is incremental as it builds on existing multimodal and acoustic modeling approaches.
The paper tackles the problem of generating acoustic profiles for indoor scenes based on user-defined material configurations, introducing a novel encoder-decoder model that outperforms baselines and state-of-the-art methods in producing high-fidelity Room Impulse Responses.
How would the sound in a studio change with a carpeted floor and acoustic tiles on the walls? We introduce the task of material-controlled acoustic profile generation, where, given an indoor scene with specific audio-visual characteristics, the goal is to generate a target acoustic profile based on a user-defined material configuration at inference time. We address this task with a novel encoder-decoder approach that encodes the scene's key properties from an audio-visual observation and generates the target Room Impulse Response (RIR) conditioned on the material specifications provided by the user. Our model enables the generation of diverse RIRs based on various material configurations defined dynamically at inference time. To support this task, we create a new benchmark, the Acoustic Wonderland Dataset, designed for developing and evaluating material-aware RIR prediction methods under diverse and challenging settings. Our results demonstrate that the proposed model effectively encodes material information and generates high-fidelity RIRs, outperforming several baselines and state-of-the-art methods.