Benchmarking and Learning Multi-Dimensional Quality Evaluator for Text-to-3D Generation
This addresses evaluation challenges for researchers and developers in text-to-3D generation, though it is incremental as it builds on existing evaluation frameworks.
The authors tackled the lack of fine-grained and multi-dimensional evaluation in text-to-3D generation by proposing MATE-3D, a benchmark with 1,280 meshes and 107,520 annotations, and HyperScore, a quality evaluator that outperforms existing metrics.
Text-to-3D generation has achieved remarkable progress in recent years, yet evaluating these methods remains challenging for two reasons: i) Existing benchmarks lack fine-grained evaluation on different prompt categories and evaluation dimensions. ii) Previous evaluation metrics only focus on a single aspect (e.g., text-3D alignment) and fail to perform multi-dimensional quality assessment. To address these problems, we first propose a comprehensive benchmark named MATE-3D. The benchmark contains eight well-designed prompt categories that cover single and multiple object generation, resulting in 1,280 generated textured meshes. We have conducted a large-scale subjective experiment from four different evaluation dimensions and collected 107,520 annotations, followed by detailed analyses of the results. Based on MATE-3D, we propose a novel quality evaluator named HyperScore. Utilizing hypernetwork to generate specified mapping functions for each evaluation dimension, our metric can effectively perform multi-dimensional quality assessment. HyperScore presents superior performance over existing metrics on MATE-3D, making it a promising metric for assessing and improving text-to-3D generation. The project is available at https://mate-3d.github.io/.