Weakly Supervised Face Naming with Symmetry-Enhanced Contrastive Loss
This work addresses the problem of automatically labeling faces with names from captions in images for applications like news understanding, representing an incremental improvement over existing methods.
The paper tackles the weakly supervised cross-modal face-name alignment task by proposing SECLA and SECLA-B models, which use symmetry-enhanced contrastive loss and a two-stage bootstrapping approach to maximize similarity scores between faces and names, achieving state-of-the-art results on datasets like Labeled Faces in the Wild and Celebrity Together.
We revisit the weakly supervised cross-modal face-name alignment task; that is, given an image and a caption, we label the faces in the image with the names occurring in the caption. Whereas past approaches have learned the latent alignment between names and faces by uncertainty reasoning over a set of images and their respective captions, in this paper, we rely on appropriate loss functions to learn the alignments in a neural network setting and propose SECLA and SECLA-B. SECLA is a Symmetry-Enhanced Contrastive Learning-based Alignment model that can effectively maximize the similarity scores between corresponding faces and names in a weakly supervised fashion. A variation of the model, SECLA-B, learns to align names and faces as humans do, that is, learning from easy to hard cases to further increase the performance of SECLA. More specifically, SECLA-B applies a two-stage learning framework: (1) Training the model on an easy subset with a few names and faces in each image-caption pair. (2) Leveraging the known pairs of names and faces from the easy cases using a bootstrapping strategy with additional loss to prevent forgetting and learning new alignments at the same time. We achieve state-of-the-art results for both the augmented Labeled Faces in the Wild dataset and the Celebrity Together dataset. In addition, we believe that our methods can be adapted to other multimodal news understanding tasks.