Contextual Metric Meta-Evaluation by Measuring Local Metric Accuracy
This work addresses the need for more precise metric evaluation in NLP, which is crucial for benchmarking systems in specific applications, though it is incremental by adapting existing meta-evaluation approaches to context-specific scenarios.
The paper tackles the problem of meta-evaluating automatic evaluation metrics in NLP by focusing on contextual settings rather than global assessments, demonstrating that metric accuracy varies significantly across different contexts such as translation and speech recognition tasks.
Meta-evaluation of automatic evaluation metrics -- assessing evaluation metrics themselves -- is crucial for accurately benchmarking natural language processing systems and has implications for scientific inquiry, production model development, and policy enforcement. While existing approaches to metric meta-evaluation focus on general statements about the absolute and relative quality of metrics across arbitrary system outputs, in practice, metrics are applied in highly contextual settings, often measuring the performance for a highly constrained set of system outputs. For example, we may only be interested in evaluating a specific model or class of models. We introduce a method for contextual metric meta-evaluation by comparing the local metric accuracy of evaluation metrics. Across translation, speech recognition, and ranking tasks, we demonstrate that the local metric accuracies vary both in absolute value and relative effectiveness as we shift across evaluation contexts. This observed variation highlights the importance of adopting context-specific metric evaluations over global ones.