Latent Mappings: Generating Open-Ended Expressive Mappings Using Variational Autoencoders
This addresses the need for open-ended mapping tools for artists and creators in interactive systems, offering a novel approach but with incremental technical contributions.
The paper tackles the problem of creating expressive gestural mappings without predefined input/output pairs by introducing latent mappings using Variational Autoencoders on unlabeled data, demonstrated in a system called Sonified Body with dancers.
In many contexts, creating mappings for gestural interactions can form part of an artistic process. Creators seeking a mapping that is expressive, novel, and affords them a sense of authorship may not know how to program it up in a signal processing patch. Tools like Wekinator and MIMIC allow creators to use supervised machine learning to learn mappings from example input/output pairings. However, a creator may know a good mapping when they encounter it yet start with little sense of what the inputs or outputs should be. We call this an open-ended mapping process. Addressing this need, we introduce the latent mapping, which leverages the latent space of an unsupervised machine learning algorithm such as a Variational Autoencoder trained on a corpus of unlabelled gestural data from the creator. We illustrate it with Sonified Body, a system mapping full-body movement to sound which we explore in a residency with three dancers.