Sonification of Facial Actions for Musical Expression
This addresses the need for novel interfaces in musical expression, but it is incremental as it builds on existing face detection and sound synthesis techniques.
The paper tackled the problem of using facial actions for musical expression by designing a system that maps facial movements to sound synthesis, reporting on gesture-to-sound mappings and preliminary performance experiments.
The central role of the face in social interaction and non-verbal communication suggests we explore facial action as a means of musical expression. This paper presents the design, implementation, and preliminary studies of a novel system utilizing face detection and optic flow algorithms to associate facial movements with sound synthesis in a topographically specific fashion. We report on our experience with various gesture-to-sound mappings and applications, and describe our preliminary experiments at musical performance using the system.