Challenges learning from imbalanced data using tree-based models: Prevalence estimates systematically depend on hyperparameters and can be upwardly biased
This addresses a critical issue for practitioners using tree-based models on imbalanced data, revealing systematic biases that affect prevalence estimation, though it is incremental in nature.
The paper tackles the problem of biased prevalence estimates in imbalanced binary classification when using random forests with subsampling and analytical calibration, showing that these estimates depend on hyperparameters and can be upwardly biased, with a surprising discovery that decision trees can be biased towards the minority class.
When using machine learning for imbalanced binary classification problems, it is common to subsample the majority class to create a (more) balanced training dataset. This biases the model's predictions because the model learns from data whose data generating process differs from new data. One way of accounting for this bias is analytically mapping the resulting predictions to new values based on the sampling rate for the majority class. We show that calibrating a random forest this way has negative consequences, including prevalence estimates that depend on both the number of predictors considered at each split in the random forest and the sampling rate used. We explain the former using known properties of random forests and analytical calibration. Through investigating the latter issue, we made a surprising discovery - contrary to the widespread belief that decision trees are biased towards the majority class, they actually can be biased towards the minority class.