Differentiable Digital Signal Processing Mixture Model for Synthesis Parameter Extraction from Mixture of Harmonic Sounds
This work addresses a domain-specific problem for audio synthesis and music processing by enabling parameter extraction from polyphonic harmonic mixtures, representing an incremental improvement over monophonic methods.
The paper tackles the problem of extracting synthesis parameters from mixtures of harmonic sounds, which existing DDSP autoencoders cannot handle, by proposing a DDSP mixture model that sums outputs of multiple pretrained autoencoders and fitting it to observed mixtures, resulting in high and stable performance compared to a baseline method.
A differentiable digital signal processing (DDSP) autoencoder is a musical sound synthesizer that combines a deep neural network (DNN) and spectral modeling synthesis. It allows us to flexibly edit sounds by changing the fundamental frequency, timbre feature, and loudness (synthesis parameters) extracted from an input sound. However, it is designed for a monophonic harmonic sound and cannot handle mixtures of harmonic sounds. In this paper, we propose a model (DDSP mixture model) that represents a mixture as the sum of the outputs of multiple pretrained DDSP autoencoders. By fitting the output of the proposed model to the observed mixture, we can directly estimate the synthesis parameters of each source. Through synthesis parameter extraction experiments, we show that the proposed method has high and stable performance compared with a straightforward method that applies the DDSP autoencoder to the signals separated by an audio source separation method.