GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
This addresses the challenge of automated evaluation for video generation, which lacks semantic understanding, but is incremental as it builds on existing datasets and models.
The paper tackles the problem of evaluating text-to-video generation by introducing GRADEO, a model that uses multi-step reasoning to provide explainable scores, showing it aligns better with human evaluations than existing methods.
Recent great advances in video generation models have demonstrated their potential to produce high-quality videos, bringing challenges to effective evaluation. Unlike human evaluation, existing automated evaluation metrics lack high-level semantic understanding and reasoning capabilities for video, thus making them infeasible and unexplainable. To fill this gap, we curate GRADEO-Instruct, a multi-dimensional T2V evaluation instruction tuning dataset, including 3.3k videos from over 10 existing video generation models and multi-step reasoning assessments converted by 16k human annotations. We then introduce GRADEO, one of the first specifically designed video evaluation models, which grades AI-generated videos for explainable scores and assessments through multi-step reasoning. Experiments show that our method aligns better with human evaluations than existing methods. Furthermore, our benchmarking reveals that current video generation models struggle to produce content that aligns with human reasoning and complex real-world scenarios. The models, datasets, and codes will be released soon.