Real-Time Control of a Virtual Orchestra by Recognition of Conducting Gestures
This work provides an engaging interactive experience for museum visitors, but the approach is incremental, applying existing gesture recognition and playback control techniques to a specific application domain.
The paper presents a museum installation that allows visitors to conduct a virtual orchestra in real-time using gesture recognition, achieving a timing accuracy of 0.2 seconds and high user satisfaction in usability studies.
We present a museum installation in a 180° dome theater, which gives the museum visitor the experience of conducting a symphony orchestra. We have pre-recorded a short music piece performed by a professional orchestra. This recording is played back in the dome with the visitor standing in the conductor's position. The visitor's gestures are captured with a vision-based skeleton tracker, steering the recording playback pace via a gesture recognition module that translates the gestures into a time control signal. This is sent to a playback module that plays the recording in the dome at the corresponding speed. The gesture recognition module is based on a hierarchical LSTM network, trained with recorded sequences of multiple conductors with different level of expertise conducting the same recording. The system is evaluated with a quantitative study of the estimated timing accuracy, a user study evaluating the musical realism and usability of the real-time control, and a field study to evaluate the performance of the entire system with real museum visitors.