Using Active Speaker Faces for Diarization in TV shows
This addresses the problem of accurate speaker diarization for media analysis in TV shows, offering a novel visual-based approach that is incremental over existing audio methods.
The paper tackles speaker diarization in TV shows by using active speaker faces for clustering, achieving superior performance compared to state-of-the-art audio-based methods, with analysis showing that even moderately performing active speaker systems can outperform audio-based ones.
Speaker diarization is one of the critical components of computational media intelligence as it enables a character-level analysis of story portrayals and media content understanding. Automated audio-based speaker diarization of entertainment media poses challenges due to the diverse acoustic conditions present in media content, be it background music, overlapping speakers, or sound effects. At the same time, speaking faces in the visual modality provide complementary information and not prone to the errors seen in the audio modality. In this paper, we address the problem of speaker diarization in TV shows using the active speaker faces. We perform face clustering on the active speaker faces and show superior speaker diarization performance compared to the state-of-the-art audio-based diarization methods. We additionally report a systematic analysis of the impact of active speaker face detection quality on the diarization performance. We also observe that a moderately well-performing active speaker system could outperform the audio-based diarization systems.