EvalTalker: Learning to Evaluate Real-Portrait-Driven Multi-Subject Talking Humans
This addresses the need for better evaluation in multi-subject talking human generation, which is incremental as it builds on existing multi-talker methods by providing a dataset and assessment tool.
The paper tackles the problem of quality degradation in multi-subject talking human generation by constructing THQA-MT, a dataset of 5,492 multi-talker-generated talking humans, and introducing EvalTalker, a quality assessment framework that achieves superior correlation with subjective scores.
Speech-driven Talking Human (TH) generation, commonly known as "Talker," currently faces limitations in multi-subject driving capabilities. Extending this paradigm to "Multi-Talker," capable of animating multiple subjects simultaneously, introduces richer interactivity and stronger immersion in audiovisual communication. However, current Multi-Talkers still exhibit noticeable quality degradation caused by technical limitations, resulting in suboptimal user experiences. To address this challenge, we construct THQA-MT, the first large-scale Multi-Talker-generated Talking Human Quality Assessment dataset, consisting of 5,492 Multi-Talker-generated THs (MTHs) from 15 representative Multi-Talkers using 400 real portraits collected online. Through subjective experiments, we analyze perceptual discrepancies among different Multi-Talkers and identify 12 common types of distortion. Furthermore, we introduce EvalTalker, a novel TH quality assessment framework. This framework possesses the ability to perceive global quality, human characteristics, and identity consistency, while integrating Qwen-Sync to perceive multimodal synchrony. Experimental results demonstrate that EvalTalker achieves superior correlation with subjective scores, providing a robust foundation for future research on high-quality Multi-Talker generation and evaluation.