Mixture of Dynamical Variational Autoencoders for Multi-Source Trajectory Modeling and Separation
This work addresses trajectory modeling and separation for applications like computer vision and audio processing, but it is incremental as it builds on existing variational autoencoder methods.
The paper tackles the problem of modeling and separating trajectories from multiple moving sources by proposing MixDVAE, a latent-variable generative model that integrates pre-trained dynamical variational autoencoders into a mixture framework, and it outperforms baseline methods in multi-object tracking and audio source separation tasks.
In this paper, we propose a latent-variable generative model called mixture of dynamical variational autoencoders (MixDVAE) to model the dynamics of a system composed of multiple moving sources. A DVAE model is pre-trained on a single-source dataset to capture the source dynamics. Then, multiple instances of the pre-trained DVAE model are integrated into a multi-source mixture model with a discrete observation-to-source assignment latent variable. The posterior distributions of both the discrete observation-to-source assignment variable and the continuous DVAE variables representing the sources content/position are estimated using a variational expectation-maximization algorithm, leading to multi-source trajectories estimation. We illustrate the versatility of the proposed MixDVAE model on two tasks: a computer vision task, namely multi-object tracking, and an audio processing task, namely single-channel audio source separation. Experimental results show that the proposed method works well on these two tasks, and outperforms several baseline methods.