EgoSonics: Generating Synchronized Audio for Silent Egocentric Videos
This addresses the need for realistic audio generation in applications like virtual reality and assistive technologies, representing a novel domain-specific advancement beyond existing limited audio synthesis methods.
The paper tackles the problem of generating synchronized audio for silent egocentric videos, achieving state-of-the-art performance in audio quality and synchronization through a novel method based on latent diffusion models and ControlNet.
We introduce EgoSonics, a method to generate semantically meaningful and synchronized audio tracks conditioned on silent egocentric videos. Generating audio for silent egocentric videos could open new applications in virtual reality, assistive technologies, or for augmenting existing datasets. Existing work has been limited to domains like speech, music, or impact sounds and cannot capture the broad range of audio frequencies found in egocentric videos. EgoSonics addresses these limitations by building on the strengths of latent diffusion models for conditioned audio synthesis. We first encode and process paired audio-video data to make them suitable for generation. The encoded data is then used to train a model that can generate an audio track that captures the semantics of the input video. Our proposed SyncroNet builds on top of ControlNet to provide control signals that enables generation of temporally synchronized audio. Extensive evaluations and a comprehensive user study show that our model outperforms existing work in audio quality, and in our proposed synchronization evaluation method. Furthermore, we demonstrate downstream applications of our model in improving video summarization.