THEval. Evaluation Framework for Talking Head Video Generation
This provides a standardized evaluation framework for researchers and developers in video generation to benchmark and improve methods, though it is incremental as it builds on existing metrics.
The paper tackles the lack of adequate evaluation metrics for talking head video generation by proposing a new framework with 8 metrics across quality, naturalness, and synchronization dimensions, based on experiments with 85,000 videos from 17 models, revealing challenges in expressiveness and artifact-free details despite good lip synchronization.
Video generation has achieved remarkable progress, with generated videos increasingly resembling real ones. However, the rapid advance in generation has outpaced the development of adequate evaluation metrics. Currently, the assessment of talking head generation primarily relies on limited metrics, evaluating general video quality, lip synchronization, and on conducting user studies. Motivated by this, we propose a new evaluation framework comprising 8 metrics related to three dimensions (i) quality, (ii) naturalness, and (iii) synchronization. In selecting the metrics, we place emphasis on efficiency, as well as alignment with human preferences. Based on this considerations, we streamline to analyze fine-grained dynamics of head, mouth, and eyebrows, as well as face quality. Our extensive experiments on 85,000 videos generated by 17 state-of-the-art models suggest that while many algorithms excel in lip synchronization, they face challenges with generating expressiveness and artifact-free details. These videos were generated based on a novel real dataset, that we have curated, in order to mitigate bias of training data. Our proposed benchmark framework is aimed at evaluating the improvement of generative methods. Original code, dataset and leaderboards will be publicly released and regularly updated with new methods, in order to reflect progress in the field.