Disentanglement of Sources in a Multi-Stream Variational Autoencoder
This work addresses the challenge of source separation in domains like acoustics and image processing, offering a domain-agnostic method that is incremental as it builds on existing VAE frameworks with a novel combination approach.
The paper tackled the problem of learning disentangled representations by proposing a multi-stream VAE (MS-VAE) that uses discrete latents to combine VAE-representations of individual sources, achieving clear separation of superimposed hand-written digits and a low rate of missed speakers in speaker diarization tasks.
Variational autoencoders (VAEs) are a leading approach to address the problem of learning disentangled representations. Typically a single VAE is used and disentangled representations are sought in its continuous latent space. Here we explore a different approach by using discrete latents to combine VAE-representations of individual sources. The combination is done based on an explicit model for source combination, and we here use a linear combination model which is well suited, e.g., for acoustic data. We formally define such a multi-stream VAE (MS-VAE) approach, derive its inference and learning equations, and we numerically investigate its principled functionality. The MS-VAE is domain-agnostic, and we here explore its ability to separate sources into different streams using superimposed hand-written digits, and mixed acoustic sources in a speaker diarization task. We observe a clear separation of digits, and on speaker diarization we observe an especially low rate of missed speakers. Numerical experiments further highlight the flexibility of the approach across varying amounts of supervision and training data.