Macro F1 and Macro F1
This clarifies a metric inconsistency for researchers and practitioners in machine learning, but it is incremental as it addresses a specific calculation issue.
The paper identifies two different formulas for calculating the macro F1 metric in classification problems, showing they are rarely equivalent and can differ by up to 0.5, potentially leading to different classifier rankings.
The 'macro F1' metric is frequently used to evaluate binary, multi-class and multi-label classification problems. Yet, we find that there exist two different formulas to calculate this quantity. In this note, we show that only under rare circumstances the two computations can be considered equivalent. More specifically, one formula well 'rewards' classifiers which produce a skewed error type distribution. In fact, the difference in outcome of the two computations can be as high as 0.5. The two computations may not only diverge in their scalar result but can also lead to different classifier rankings.