Sound-Stream II: Towards Real-Time Gesture Controlled Articulatory Sound Synthesis
This work addresses the problem of intuitive, real-time sound synthesis for applications in speech research and interactive media, though it appears incremental as it builds on existing biomechanical and synthesis toolkits.
The paper tackles real-time gesture-controlled articulatory sound synthesis by developing an interface that uses four force-based sensors to control a 2D tongue model in ArtiSynth, mapping gestures to sound via JASS synthesis, enabling users to produce vocal sounds through physical interaction.
We present an interface involving four degrees-of-freedom (DOF) mechanical control of a two dimensional, mid-sagittal tongue through a biomechanical toolkit called ArtiSynth and a sound synthesis engine called JASS towards articulatory sound synthesis. As a demonstration of the project, the user will learn to produce a range of JASS vocal sounds, by varying the shape and position of the ArtiSynth tongue in 2D space through a set of four force-based sensors. In other words, the user will be able to physically play around with these four sensors, thereby virtually controlling the magnitude of four selected muscle excitations of the tongue to vary articulatory structure. This variation is computed in terms of Area Functions in ArtiSynth environment and communicated to the JASS based audio-synthesizer coupled with two-mass glottal excitation model to complete this end-to-end gesture-to-sound mapping.