Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis
This work addresses the challenge of simulating acoustic environments for applications like audio processing and virtual reality, offering a novel cross-modal synthesis approach that is incremental in applying neural networks to a specific domain.
The paper tackles the problem of generating acoustic impulse responses (IRs) from single images to simulate reverberation, eliminating the need for time-intensive and expensive physical recordings. It demonstrates plausible IR synthesis across diverse settings, including real-world locations, paintings, animations, and synthetic environments, with evaluation based on comparisons to ground truth and human expert assessment.
Measuring the acoustic characteristics of a space is often done by capturing its impulse response (IR), a representation of how a full-range stimulus sound excites it. This work generates an IR from a single image, which can then be applied to other signals using convolution, simulating the reverberant characteristics of the space shown in the image. Recording these IRs is both time-intensive and expensive, and often infeasible for inaccessible locations. We use an end-to-end neural network architecture to generate plausible audio impulse responses from single images of acoustic environments. We evaluate our method both by comparisons to ground truth data and by human expert evaluation. We demonstrate our approach by generating plausible impulse responses from diverse settings and formats including well known places, musical halls, rooms in paintings, images from animations and computer games, synthetic environments generated from text, panoramic images, and video conference backgrounds.