Audio inpainting with generative adversarial network
This work addresses audio inpainting for music production or restoration, but it is incremental as it builds on existing WGAN methods with architectural modifications.
The authors tackled the problem of audio inpainting for long-range gaps (500 ms) using a new Wasserstein Generative Adversarial Network (WGAN) architecture that incorporates short-range and long-range neighboring borders, outperforming a classical WGAN model by improving reconstruction of high-frequency content and achieving better objective difference grading (ODG) scores, particularly for instruments like piano and guitar with lower frequency spectra.
We study the ability of Wasserstein Generative Adversarial Network (WGAN) to generate missing audio content which is, in context, (statistically similar) to the sound and the neighboring borders. We deal with the challenge of audio inpainting long range gaps (500 ms) using WGAN models. We improved the quality of the inpainting part using a new proposed WGAN architecture that uses a short-range and a long-range neighboring borders compared to the classical WGAN model. The performance was compared with two different audio instruments (piano and guitar) and on virtuoso pianists together with a string orchestra. The objective difference grading (ODG) was used to evaluate the performance of both architectures. The proposed model outperforms the classical WGAN model and improves the reconstruction of high-frequency content. Further, we got better results for instruments where the frequency spectrum is mainly in the lower range where small noises are less annoying for human ear and the inpainting part is more perceptible. Finally, we could show that better test results for audio dataset were reached where a particular instrument is accompanist by other instruments if we train the network only on this particular instrument neglecting the other instruments.