FaceLLM: A Multimodal Large Language Model for Face Understanding
This work addresses the need for better facial understanding in AI systems, particularly for tasks involving expression, emotion, and demographic features, though it is incremental as it builds on existing MLLM frameworks with domain-specific data.
The paper tackles the problem of multimodal large language models (MLLMs) lacking domain-specific reasoning for facial images by introducing FaceLLM, a model trained on a novel dataset called FairFaceGPT, which improves performance on face-centric tasks and achieves state-of-the-art results.
Multimodal large language models (MLLMs) have shown remarkable performance in vision-language tasks. However, existing MLLMs are primarily trained on generic datasets, limiting their ability to reason on domain-specific visual cues such as those in facial images. In particular, tasks that require detailed understanding of facial structure, expression, emotion, and demographic features remain underexplored by MLLMs due to the lack of large-scale annotated face image-text datasets. In this work, we introduce FaceLLM, a multimodal large language model trained specifically for facial image understanding. To construct the training data, we propose a novel weakly supervised pipeline that uses ChatGPT with attribute-aware prompts to generate high-quality question-answer pairs based on images from the FairFace dataset. The resulting corpus, called FairFaceGPT, covers a diverse set of attributes including expression, pose, skin texture, and forensic information. Our experiments demonstrate that FaceLLM improves the performance of MLLMs on various face-centric tasks and achieves state-of-the-art performance. This work highlights the potential of synthetic supervision via language models for building domain-specialized MLLMs, and sets a precedent for trustworthy, human-centric multimodal AI systems. FairFaceGPT dataset and pretrained FaceLLM models are publicly available in the project page.