We Need to Talk About Classification Evaluation Metrics in NLP
This addresses the problem of inconsistent evaluation in NLP for researchers and practitioners, though it is incremental as it builds on existing metric analysis.
The paper tackles the lack of consensus on evaluation metrics in NLP classification tasks by comparing standard and exotic metrics, finding that a random-guess normalized Informedness metric best captures ideal model characteristics across various tasks.
In Natural Language Processing (NLP) classification tasks such as topic categorisation and sentiment analysis, model generalizability is generally measured with standard metrics such as Accuracy, F-Measure, or AUC-ROC. The diversity of metrics, and the arbitrariness of their application suggest that there is no agreement within NLP on a single best metric to use. This lack suggests there has not been sufficient examination of the underlying heuristics which each metric encodes. To address this we compare several standard classification metrics with more 'exotic' metrics and demonstrate that a random-guess normalised Informedness metric is a parsimonious baseline for task performance. To show how important the choice of metric is, we perform extensive experiments on a wide range of NLP tasks including a synthetic scenario, natural language understanding, question answering and machine translation. Across these tasks we use a superset of metrics to rank models and find that Informedness best captures the ideal model characteristics. Finally, we release a Python implementation of Informedness following the SciKitLearn classifier format.