What is the Best Automated Metric for Text to Motion Generation?
This addresses a critical evaluation gap for researchers in text-to-motion generation, providing a more reliable automated metric to replace human evaluations, though it is incremental as it builds on existing metric analysis.
The paper tackled the problem of evaluating text-to-motion generation by systematically studying which automated metrics best align with human judgments, finding that current metrics show poor sample-level correlation but some work well for model-level assessment, and proposed a new metric, MoBERT, that offers strong human-correlated sample-level evaluations with near-perfect model-level correlation.
There is growing interest in generating skeleton-based human motions from natural language descriptions. While most efforts have focused on developing better neural architectures for this task, there has been no significant work on determining the proper evaluation metric. Human evaluation is the ultimate accuracy measure for this task, and automated metrics should correlate well with human quality judgments. Since descriptions are compatible with many motions, determining the right metric is critical for evaluating and designing effective generative models. This paper systematically studies which metrics best align with human evaluations and proposes new metrics that align even better. Our findings indicate that none of the metrics currently used for this task show even a moderate correlation with human judgments on a sample level. However, for assessing average model performance, commonly used metrics such as R-Precision and less-used coordinate errors show strong correlations. Additionally, several recently developed metrics are not recommended due to their low correlation compared to alternatives. We also introduce a novel metric based on a multimodal BERT-like model, MoBERT, which offers strongly human-correlated sample-level evaluations while maintaining near-perfect model-level correlation. Our results demonstrate that this new metric exhibits extensive benefits over all current alternatives.