Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation
This work addresses the data scarcity and alignment issues in audio-to-image generation for researchers and practitioners in multimodal AI, though it is incremental as it builds on existing retrieval and generative techniques.
The authors tackled the problem of training audio-to-image generative models by proposing a scalable image sonification framework that artificially pairs high-quality uni-modal data using vision-language models, eliminating the need for ground truth audio-visual pairs. Their model performs competitively against state-of-the-art methods and implicitly develops auditory capabilities like semantic mixing and interpolation.
Training audio-to-image generative models requires an abundance of diverse audio-visual pairs that are semantically aligned. Such data is almost always curated from in-the-wild videos, given the cross-modal semantic correspondence that is inherent to them. In this work, we hypothesize that insisting on the absolute need for ground truth audio-visual correspondence, is not only unnecessary, but also leads to severe restrictions in scale, quality, and diversity of the data, ultimately impairing its use in the modern generative models. That is, we propose a scalable image sonification framework where instances from a variety of high-quality yet disjoint uni-modal origins can be artificially paired through a retrieval process that is empowered by reasoning capabilities of modern vision-language models. To demonstrate the efficacy of this approach, we use our sonified images to train an audio-to-image generative model that performs competitively against state-of-the-art. Finally, through a series of ablation studies, we exhibit several intriguing auditory capabilities like semantic mixing and interpolation, loudness calibration and acoustic space modeling through reverberation that our model has implicitly developed to guide the image generation process.