MelNet: A Generative Model for Audio in the Frequency Domain
This work addresses the problem of generating structured audio for applications in speech and music synthesis, showing incremental improvements over existing methods.
The authors tackled the challenge of modeling long-range dependencies in audio generation by using a time-frequency representation, resulting in a model that produces high-fidelity audio samples with improved density estimates and human judgments across tasks like speech and music generation.
Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.