CAESynth: Real-Time Timbre Interpolation and Pitch Control with Conditional Autoencoders
This work addresses real-time audio synthesis for musicians and mixed reality developers, offering a novel method for timbre interpolation and pitch control, though it is incremental in its approach.
The authors tackled real-time audio synthesis by developing CAESynth, a conditional autoencoder that interpolates timbre in a latent space while controlling pitch independently, achieving smooth and high-fidelity synthesis for musical cues and mixed reality applications.
In this paper, we present a novel audio synthesizer, CAESynth, based on a conditional autoencoder. CAESynth synthesizes timbre in real-time by interpolating the reference sounds in their shared latent feature space, while controlling a pitch independently. We show that training a conditional autoencoder based on accuracy in timbre classification together with adversarial regularization of pitch content allows timbre distribution in latent space to be more effective and stable for timbre interpolation and pitch conditioning. The proposed method is applicable not only to creation of musical cues but also to exploration of audio affordance in mixed reality based on novel timbre mixtures with environmental sounds. We demonstrate by experiments that CAESynth achieves smooth and high-fidelity audio synthesis in real-time through timbre interpolation and independent yet accurate pitch control for musical cues as well as for audio affordance with environmental sound. A Python implementation along with some generated samples are shared online.