On the Calibration of Nested Dichotomies for Large Multiclass Tasks
This addresses calibration issues in nested dichotomies for multiclass tasks, which is incremental but important for practical applications.
The paper tackles the problem of poor probability calibration in nested dichotomies for multiclass classification, showing that calibration strategies significantly improve accuracy and log-loss, especially with many classes.
Nested dichotomies are used as a method of transforming a multiclass classification problem into a series of binary problems. A tree structure is induced that recursively splits the set of classes into subsets, and a binary classification model learns to discriminate between the two subsets of classes at each node. In this paper, we demonstrate that these nested dichotomies typically exhibit poor probability calibration, even when the base binary models are well calibrated. We also show that this problem is exacerbated when the binary models are poorly calibrated. We discuss the effectiveness of different calibration strategies and show that accuracy and log-loss can be significantly improved by calibrating both the internal base models and the full nested dichotomy structure, especially when the number of classes is high.