MAAS: Multi-modal Assignation for Active Speaker Detection
This work provides a strong specific gain for researchers and developers working on active speaker detection by improving performance on a standard benchmark.
This paper tackles the problem of active speaker detection by proposing a multi-modal assignation strategy that links visual features of potential speakers to detected speech events. Their method achieves a new state-of-the-art mAP of 88.8% on the AVA-ActiveSpeaker dataset.
Active speaker detection requires a solid integration of multi-modal cues. While individual modalities can approximate a solution, accurate predictions can only be achieved by explicitly fusing the audio and visual features and modeling their temporal progression. Despite its inherent muti-modal nature, current methods still focus on modeling and fusing short-term audiovisual features for individual speakers, often at frame level. In this paper we present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem, and provides a straightforward strategy where independent visual features from potential speakers in the scene are assigned to a previously detected speech event. Our experiments show that, an small graph data structure built from a single frame, allows to approximate an instantaneous audio-visual assignment problem. Moreover, the temporal extension of this initial graph achieves a new state-of-the-art on the AVA-ActiveSpeaker dataset with a mAP of 88.8\%.