Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora
This work addresses the lack of standard evaluation methods for corpus-level semantic metrics in NLP, providing tools for researchers and practitioners to compare metric behaviors, though it is incremental as it builds on existing metrics.
The paper tackled the problem of evaluating semantic similarity metrics for text corpora by proposing automatic and interpretable measures, revealing that recent metrics better identify semantic mismatches while classical ones are more sensitive to surface-level perturbations.
The ability to compare the semantic similarity between text corpora is important in a variety of natural language processing applications. However, standard methods for evaluating these metrics have yet to be established. We propose a set of automatic and interpretable measures for assessing the characteristics of corpus-level semantic similarity metrics, allowing sensible comparison of their behavior. We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by evaluating them on a collection of classical and state-of-the-art metrics. Our measures revealed that recently-developed metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.