Speaker Recognition in Realistic Scenario Using Multimodal Data
This work addresses speaker recognition in realistic scenarios for applications like security or media, but it is incremental as it builds on existing multimodal methods.
The paper tackled speaker recognition by leveraging large-scale audio-visual data from YouTube, proposing a two-branch network to learn joint face-voice representations, and found that adding facial information improved performance on the VoxCeleb1 dataset.
In recent years, an association is established between faces and voices of celebrities leveraging large scale audio-visual information from YouTube. The availability of large scale audio-visual datasets is instrumental in developing speaker recognition methods based on standard Convolutional Neural Networks. Thus, the aim of this paper is to leverage large scale audio-visual information to improve speaker recognition task. To achieve this task, we proposed a two-branch network to learn joint representations of faces and voices in a multimodal system. Afterwards, features are extracted from the two-branch network to train a classifier for speaker recognition. We evaluated our proposed framework on a large scale audio-visual dataset named VoxCeleb$1$. Our results show that addition of facial information improved the performance of speaker recognition. Moreover, our results indicate that there is an overlap between face and voice.