Fuse after Align: Improving Face-Voice Association Learning via Multimodal Encoder
This work addresses the problem of improving voice-face association learning for retrieval and matching tasks, representing an incremental advance over existing methods.
The paper tackles the problem of learning voice-face associations by introducing a novel unsupervised framework that uses a multimodal encoder after contrastive learning and binary classification, achieving state-of-the-art results with improvements of approximately 3% in verification, 2.5% in matching, and 1.3% in retrieval.
Today, there have been many achievements in learning the association between voice and face. However, most previous work models rely on cosine similarity or L2 distance to evaluate the likeness of voices and faces following contrastive learning, subsequently applied to retrieval and matching tasks. This method only considers the embeddings as high-dimensional vectors, utilizing a minimal scope of available information. This paper introduces a novel framework within an unsupervised setting for learning voice-face associations. By employing a multimodal encoder after contrastive learning and addressing the problem through binary classification, we can learn the implicit information within the embeddings in a more effective and varied manner. Furthermore, by introducing an effective pair selection method, we enhance the learning outcomes of both contrastive learning and the matching task. Empirical evidence demonstrates that our framework achieves state-of-the-art results in voice-face matching, verification, and retrieval tasks, improving verification by approximately 3%, matching by about 2.5%, and retrieval by around 1.3%.