ImmerseDiffusion: A Generative Spatial Audio Latent Diffusion Model
This work addresses the need for controllable spatial audio generation for applications in virtual reality, gaming, and multimedia, representing an incremental advancement by combining existing diffusion and encoding techniques in a novel domain-specific setup.
The paper tackles the problem of generating 3D immersive soundscapes by introducing ImmerseDiffusion, an end-to-end generative audio model that produces first-order ambisonics audio conditioned on spatial, temporal, and environmental inputs, with evaluations showing promising results in generation quality and spatial fidelity.
We introduce ImmerseDiffusion, an end-to-end generative audio model that produces 3D immersive soundscapes conditioned on the spatial, temporal, and environmental conditions of sound objects. ImmerseDiffusion is trained to generate first-order ambisonics (FOA) audio, which is a conventional spatial audio format comprising four channels that can be rendered to multichannel spatial output. The proposed generative system is composed of a spatial audio codec that maps FOA audio to latent components, a latent diffusion model trained based on various user input types, namely, text prompts, spatial, temporal and environmental acoustic parameters, and optionally a spatial audio and text encoder trained in a Contrastive Language and Audio Pretraining (CLAP) style. We propose metrics to evaluate the quality and spatial adherence of the generated spatial audio. Finally, we assess the model performance in terms of generation quality and spatial conformance, comparing the two proposed modes: ``descriptive", which uses spatial text prompts) and ``parametric", which uses non-spatial text prompts and spatial parameters. Our evaluations demonstrate promising results that are consistent with the user conditions and reflect reliable spatial fidelity.