SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration
This addresses usability challenges for musicians interacting with synthesizers, though it appears incremental as it builds on existing multimodal and genetic algorithm techniques.
The authors tackled the problem of complex and low-level synthesizer interfaces by developing SynthScribe, a fullstack system using multimodal deep learning for high-level sound retrieval, creation, and modification, with user studies showing reliable performance and expanded creative horizons for musicians.
Synthesizers are powerful tools that allow musicians to create dynamic and original sounds. Existing commercial interfaces for synthesizers typically require musicians to interact with complex low-level parameters or to manage large libraries of premade sounds. To address these challenges, we implement SynthScribe -- a fullstack system that uses multimodal deep learning to let users express their intentions at a much higher level. We implement features which address a number of difficulties, namely 1) searching through existing sounds, 2) creating completely new sounds, 3) making meaningful modifications to a given sound. This is achieved with three main features: a multimodal search engine for a large library of synthesizer sounds; a user centered genetic algorithm by which completely new sounds can be created and selected given the users preferences; a sound editing support feature which highlights and gives examples for key control parameters with respect to a text or audio based query. The results of our user studies show SynthScribe is capable of reliably retrieving and modifying sounds while also affording the ability to create completely new sounds that expand a musicians creative horizon.