Designing, Playing, and Performing with a Vision-based Mouth Interface
This work addresses a niche problem for musicians and performers by enabling new forms of musical expression through a vision-based mouth interface, though it appears incremental as it builds on existing computer vision and MIDI technologies.
The researchers tackled the problem of using facial actions for musical control by developing the Mouthesizer, a system that uses a camera and computer vision to extract mouth shape parameters and output them as MIDI control changes, resulting in documented experiences with gesture-to-sound mappings and a live performance.
The role of the face and mouth in speech production as well asnon-verbal communication suggests the use of facial action tocontrol musical sound. Here we document work on theMouthesizer, a system which uses a headworn miniaturecamera and computer vision algorithm to extract shapeparameters from the mouth opening and output these as MIDIcontrol changes. We report our experience with variousgesture-to-sound mappings and musical applications, anddescribe a live performance which used the Mouthesizerinterface.