Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignment
This addresses the challenge of cross-modal generation from audio to vision, which is incremental as it builds on existing methods with enhancements like sound source localization.
The paper tackles the problem of generating diverse visual scenes from in-the-wild audio by aligning audio-visual modalities and using a pre-trained image generator, achieving substantially better results on VEGAS and VGGSound datasets compared to prior work.
How does audio describe the world around us? In this work, we propose a method for generating images of visual scenes from diverse in-the-wild sounds. This cross-modal generation task is challenging due to the significant information gap between auditory and visual signals. We address this challenge by designing a model that aligns audio-visual modalities by enriching audio features with visual information and translating them into the visual latent space. These features are then fed into the pre-trained image generator to produce images. To enhance image quality, we use sound source localization to select audio-visual pairs with strong cross-modal correlations. Our method achieves substantially better results on the VEGAS and VGGSound datasets compared to previous work and demonstrates control over the generation process through simple manipulations to the input waveform or latent space. Furthermore, we analyze the geometric properties of the learned embedding space and demonstrate that our learning approach effectively aligns audio-visual signals for cross-modal generation. Based on this analysis, we show that our method is agnostic to specific design choices, showing its generalizability by integrating various model architectures and different types of audio-visual data.