Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially-Aware Language Acquisition
This addresses the need for robust active speaker detection in noisy environments to support language acquisition in artificial cognitive systems, though it is incremental as it builds on existing multimodal approaches.
The paper tackles the problem of detecting active speakers in multi-person interactions using a self-supervised visual method, achieving good performance in speaker-dependent settings but significantly lower performance in speaker-independent settings.
This paper presents a self-supervised method for visual detection of the active speaker in a multi-person spoken interaction scenario. Active speaker detection is a fundamental prerequisite for any artificial cognitive system attempting to acquire language in social settings. The proposed method is intended to complement the acoustic detection of the active speaker, thus improving the system robustness in noisy conditions. The method can detect an arbitrary number of possibly overlapping active speakers based exclusively on visual information about their face. Furthermore, the method does not rely on external annotations, thus complying with cognitive development. Instead, the method uses information from the auditory modality to support learning in the visual domain. This paper reports an extensive evaluation of the proposed method using a large multi-person face-to-face interaction dataset. The results show good performance in a speaker dependent setting. However, in a speaker independent setting the proposed method yields a significantly lower performance. We believe that the proposed method represents an essential component of any artificial cognitive system or robotic platform engaging in social interactions.