3D Arena: An Open Platform for Generative 3D Evaluation
This addresses the problem of misaligned automated metrics for generative 3D evaluation, providing a human-centered benchmark for researchers and practitioners, though it is incremental in improving evaluation methods rather than introducing a new paradigm.
The paper tackles the challenge of evaluating generative 3D models by introducing 3D Arena, an open platform that uses large-scale human preference collection via pairwise comparisons, resulting in the collection of 123,243 votes from 8,096 users across 19 models and revealing preferences such as a 16.6 ELO advantage for Gaussian splats over meshes.
Evaluating Generative 3D models remains challenging due to misalignment between automated metrics and human perception of quality. Current benchmarks rely on image-based metrics that ignore 3D structure or geometric measures that fail to capture perceptual appeal and real-world utility. To address this gap, we present 3D Arena, an open platform for evaluating image-to-3D generation models through large-scale human preference collection using pairwise comparisons. Since launching in June 2024, the platform has collected 123,243 votes from 8,096 users across 19 state-of-the-art models, establishing the largest human preference evaluation for Generative 3D. We contribute the iso3d dataset of 100 evaluation prompts and demonstrate quality control achieving 99.75% user authenticity through statistical fraud detection. Our ELO-based ranking system provides reliable model assessment, with the platform becoming an established evaluation resource. Through analysis of this preference data, we present insights into human preference patterns. Our findings reveal preferences for visual presentation features, with Gaussian splat outputs achieving a 16.6 ELO advantage over meshes and textured models receiving a 144.1 ELO advantage over untextured models. We provide recommendations for improving evaluation methods, including multi-criteria assessment, task-oriented evaluation, and format-aware comparison. The platform's community engagement establishes 3D Arena as a benchmark for the field while advancing understanding of human-centered evaluation in Generative 3D.