Interpretable Multi-dataset Evaluation for Named Entity Recognition
This provides a tool for researchers to better compare and improve NER systems, though it is incremental as it focuses on evaluation rather than new models.
The paper tackles the difficulty in understanding differences between NLP models by introducing a general methodology for interpretable evaluation in named entity recognition, enabling analysis of model and dataset differences and their interplay.
With the proliferation of models for natural language processing tasks, it is even harder to understand the differences between models and their relative merits. Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 does not tell us why or how particular methods perform differently and how diverse datasets influence the model design choices. In this paper, we present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them, identifying the strengths and weaknesses of current systems. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area: https://github.com/neulab/InterpretEval.