LMME3DHF: Benchmarking and Evaluating Multimodal 3D Human Face Generation with LMMs
This work addresses a critical problem for applications in media, VR, and security by providing a benchmark and evaluation metric for AI-generated 3D faces, though it is incremental as it builds on existing multimodal models.
The paper tackles the challenge of assessing the quality and realism of AI-generated 3D human faces by introducing Gen3DHF, a large-scale benchmark with 2,000 videos and 4,000 Mean Opinion Scores, and LMME3DHF, a multimodal model that achieves state-of-the-art performance in predicting scores and identifying distortions.
The rapid advancement in generative artificial intelligence have enabled the creation of 3D human faces (HFs) for applications including media production, virtual reality, security, healthcare, and game development, etc. However, assessing the quality and realism of these AI-generated 3D human faces remains a significant challenge due to the subjective nature of human perception and innate perceptual sensitivity to facial features. To this end, we conduct a comprehensive study on the quality assessment of AI-generated 3D human faces. We first introduce Gen3DHF, a large-scale benchmark comprising 2,000 videos of AI-Generated 3D Human Faces along with 4,000 Mean Opinion Scores (MOS) collected across two dimensions, i.e., quality and authenticity, 2,000 distortion-aware saliency maps and distortion descriptions. Based on Gen3DHF, we propose LMME3DHF, a Large Multimodal Model (LMM)-based metric for Evaluating 3DHF capable of quality and authenticity score prediction, distortion-aware visual question answering, and distortion-aware saliency prediction. Experimental results show that LMME3DHF achieves state-of-the-art performance, surpassing existing methods in both accurately predicting quality scores for AI-generated 3D human faces and effectively identifying distortion-aware salient regions and distortion types, while maintaining strong alignment with human perceptual judgments. Both the Gen3DHF database and the LMME3DHF will be released upon the publication.