A Novel Face-tracking Mouth Controller and its Application to Interacting with Bioacoustic Models
arXiv:2010.03265v124 citations
Originality Synthesis-oriented
AI Analysis
This work enables unencumbered audio control for applications like music or bioacoustics, but it is incremental as it builds on existing tracking and synthesis methods.
The authors developed a real-time face-tracking system to control audio synthesis via mouth movements, applying it to interact with a bioacoustic model of bird vocalization.
We describe a simple, computationally light, real-time system for tracking the lower face and extracting information about the shape of the open mouth from a video sequence. The system allows unencumbered control of audio synthesis modules by the action of the mouth. We report work in progress to use the mouth controller to interact with a physical model of sound production by the avian syrinx.