Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes
This work enables interactive sound generation for 3D scene reconstructions, addressing a novel domain-specific challenge in multimedia and virtual reality.
The paper tackles the problem of predicting sounds from human hand interactions in 3D scenes by training a rectified flow model on action-sound pairs, and finds that the generated sounds accurately convey material properties and are often indistinguishable from real sounds to human observers.
We study the problem of making 3D scene reconstructions interactive by asking the following question: can we predict the sounds of human hands physically interacting with a scene? First, we record a video of a human manipulating objects within a 3D scene using their hands. We then use these action-sound pairs to train a rectified flow model to map 3D hand trajectories to their corresponding audio. At test time, a user can query the model for other actions, parameterized as sequences of hand poses, to estimate their corresponding sounds. In our experiments, we find that our generated sounds accurately convey material properties and actions, and that they are often indistinguishable to human observers from real sounds. Project page: https://www.yimingdou.com/hearing_hands/