A Real-Time Active Speaker Detection System Integrating an Audio-Visual Signal with a Spatial Querying Mechanism
This work addresses the problem of efficient, real-time speaker detection for edge computing in virtual cinematography, though it is incremental as it builds on existing audio-visual methods with a novel querying approach.
The paper tackles real-time active speaker detection for meetings by introducing a neural network system that integrates audio-visual signals with a spatial querying mechanism, achieving operation with only 127 MFLOPs per participant and demonstrating graceful degradation under computational constraints.
We introduce a distinctive real-time, causal, neural network-based active speaker detection system optimized for low-power edge computing. This system drives a virtual cinematography module and is deployed on a commercial device. The system uses data originating from a microphone array and a 360-degree camera. Our network requires only 127 MFLOPs per participant, for a meeting with 14 participants. Unlike previous work, we examine the error rate of our network when the computational budget is exhausted, and find that it exhibits graceful degradation, allowing the system to operate reasonably well even in this case. Departing from conventional DOA estimation approaches, our network learns to query the available acoustic data, considering the detected head locations. We train and evaluate our algorithm on a realistic meetings dataset featuring up to 14 participants in the same meeting, overlapped speech, and other challenging scenarios.