VideoGen-Eval: Agent-based System for Video Generation Evaluation
This addresses the problem of misaligned and inflexible evaluation for video generation researchers, though it is incremental as it builds on existing agent and multimodal methods.
The paper tackles the inadequacy of existing evaluation systems for video generation by proposing VideoGen-Eval, an agent-based system that integrates LLM-based content structuring and MLLM-based content judgment, achieving strong alignment with human preferences and reliably completing evaluations, as validated by experiments with over 12,000 videos from 20+ models.
The rapid advancement of video generation has rendered existing evaluation systems inadequate for assessing state-of-the-art models, primarily due to simple prompts that cannot showcase the model's capabilities, fixed evaluation operators struggling with Out-of-Distribution (OOD) cases, and misalignment between computed metrics and human preferences. To bridge the gap, we propose VideoGen-Eval, an agent evaluation system that integrates LLM-based content structuring, MLLM-based content judgment, and patch tools designed for temporal-dense dimensions, to achieve a dynamic, flexible, and expandable video generation evaluation. Additionally, we introduce a video generation benchmark to evaluate existing cutting-edge models and verify the effectiveness of our evaluation system. It comprises 700 structured, content-rich prompts (both T2V and I2V) and over 12,000 videos generated by 20+ models, among them, 8 cutting-edge models are selected as quantitative evaluation for the agent and human. Extensive experiments validate that our proposed agent-based evaluation system demonstrates strong alignment with human preferences and reliably completes the evaluation, as well as the diversity and richness of the benchmark.