FaceXBench: Evaluating Multimodal LLMs on Face Understanding
This addresses the problem of evaluating face understanding capabilities in MLLMs for AI researchers, providing a crucial resource for development, though it is incremental as it focuses on benchmarking rather than novel model improvements.
The authors tackled the lack of systematic evaluation of multimodal large language models (MLLMs) on face understanding by introducing FaceXBench, a benchmark with 5,000 multimodal multiple-choice questions across 14 tasks, and found that current MLLMs, including advanced models like GPT-4o and GeminiPro 1.5, show significant room for improvement.
Multimodal Large Language Models (MLLMs) demonstrate impressive problem-solving abilities across a wide range of tasks and domains. However, their capacity for face understanding has not been systematically studied. To address this gap, we introduce FaceXBench, a comprehensive benchmark designed to evaluate MLLMs on complex face understanding tasks. FaceXBench includes 5,000 multimodal multiple-choice questions derived from 25 public datasets and a newly created dataset, FaceXAPI. These questions cover 14 tasks across 6 broad categories, assessing MLLMs' face understanding abilities in bias and fairness, face authentication, recognition, analysis, localization and tool retrieval. Using FaceXBench, we conduct an extensive evaluation of 26 open-source MLLMs alongside 2 proprietary models, revealing the unique challenges in complex face understanding tasks. We analyze the models across three evaluation settings: zero-shot, in-context task description, and chain-of-thought prompting. Our detailed analysis reveals that current MLLMs, including advanced models like GPT-4o, and GeminiPro 1.5, show significant room for improvement. We believe FaceXBench will be a crucial resource for developing MLLMs equipped to perform sophisticated face understanding. Code: https://github.com/Kartik-3004/facexbench