Why only Micro-F1? Class Weighting of Measures for Relation Classification
This work addresses evaluation challenges for researchers in relation classification, but it is incremental as it builds on existing weighting schemes.
The paper tackles the problem of evaluating relation classification models on imbalanced datasets by analyzing weighting schemes like micro-F1 and macro-F1, introducing a framework with new intermediate schemes, and showing that reporting multiple schemes better highlights model strengths and weaknesses.
Relation classification models are conventionally evaluated using only a single measure, e.g., micro-F1, macro-F1 or AUC. In this work, we analyze weighting schemes, such as micro and macro, for imbalanced datasets. We introduce a framework for weighting schemes, where existing schemes are extremes, and two new intermediate schemes. We show that reporting results of different weighting schemes better highlights strengths and weaknesses of a model.