SLVMEval: Synthetic Meta Evaluation Benchmark for Text-to-Long Video Generation
This addresses a fundamental need for reliable evaluation in text-to-long-video generation, which is incremental as it builds on existing datasets and methods to create a new benchmark.
The paper tackles the problem of evaluating text-to-long-video generation systems by proposing SLVMEval, a synthetic benchmark for meta-evaluating these systems on videos up to 3 hours, and finds that existing systems fall short of human accuracy in nine out of ten aspects, with human evaluators achieving 84.7%-96.8% accuracy.
This paper proposes the synthetic long-video meta-evaluation (SLVMEval), a benchmark for meta-evaluating text-to-video (T2V) evaluation systems. The proposed SLVMEval benchmark focuses on assessing these systems on videos of up to 10,486 s (approximately 3 h). The benchmark targets a fundamental requirement, namely, whether the systems can accurately assess video quality in settings that are easy for humans to assess. We adopt a pairwise comparison-based meta-evaluation framework. Building on dense video-captioning datasets, we synthetically degrade source videos to create controlled "high-quality versus low-quality" pairs across 10 distinct aspects. Then, we employ crowdsourcing to filter and retain only those pairs in which the degradation is clearly perceptible, thereby establishing an effective final testbed. Using this testbed, we assess the reliability of existing evaluation systems in ranking these pairs. Experimental results demonstrate that human evaluators can identify the better long video with 84.7%-96.8% accuracy, and in nine of the 10 aspects, the accuracy of these systems falls short of human assessment, revealing weaknesses in text-to-long-video evaluation.