An Initial Exploration: Learning to Generate Realistic Audio for Silent Video
This work addresses the problem of automating audio generation for media production, offering a potential alternative to manual Foley art, though it is incremental as it builds on existing audio generation techniques.
The paper tackled generating realistic audio for silent video using deep learning, exploring multiple architectures and finding that a transformer-based approach effectively matched low-frequency sounds to visual patterns but struggled with nuanced waveforms.
Generating realistic audio effects for movies and other media is a challenging task that is accomplished today primarily through physical techniques known as Foley art. Foley artists create sounds with common objects (e.g., boxing gloves, broken glass) in time with video as it is playing to generate captivating audio tracks. In this work, we aim to develop a deep-learning based framework that does much the same - observes video in it's natural sequence and generates realistic audio to accompany it. Notably, we have reason to believe this is achievable due to advancements in realistic audio generation techniques conditioned on other inputs (e.g., Wavenet conditioned on text). We explore several different model architectures to accomplish this task that process both previously-generated audio and video context. These include deep-fusion CNN, dilated Wavenet CNN with visual context, and transformer-based architectures. We find that the transformer-based architecture yields the most promising results, matching low-frequencies to visual patterns effectively, but failing to generate more nuanced waveforms.