Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio
This addresses the gap in modeling acoustic equivalents of 3D human bodies for applications in audio rendering and virtual environments, representing a novel domain-specific contribution.
The paper tackles the problem of modeling 3D spatial audio produced by human body motion and speech, presenting a model that generates accurate 3D sound fields from audio and body pose inputs, with empirical evaluation showing it produces accurate body-induced sound fields.
While 3D human body modeling has received much attention in computer vision, modeling the acoustic equivalent, i.e. modeling 3D spatial audio produced by body motion and speech, has fallen short in the community. To close this gap, we present a model that can generate accurate 3D spatial audio for full human bodies. The system consumes, as input, audio signals from headset microphones and body pose, and produces, as output, a 3D sound field surrounding the transmitter's body, from which spatial audio can be rendered at any arbitrary position in the 3D space. We collect a first-of-its-kind multimodal dataset of human bodies, recorded with multiple cameras and a spherical array of 345 microphones. In an empirical evaluation, we demonstrate that our model can produce accurate body-induced sound fields when trained with a suitable loss. Dataset and code are available online.