Fisher consistency for prior probability shift
This work addresses the need for trustworthy class probability estimators in machine learning applications affected by prior probability shift, though it is incremental as it builds on existing quantification methods.
The paper tackles the problem of ensuring reliable estimation of class prior probabilities under dataset shift by introducing Fisher consistency as a criterion for unbiasedness, finding that Adjusted Classify & Count and EM-algorithm are Fisher consistent while CDE-Iterate is not.
We introduce Fisher consistency in the sense of unbiasedness as a desirable property for estimators of class prior probabilities. Lack of Fisher consistency could be used as a criterion to dismiss estimators that are unlikely to deliver precise estimates in test datasets under prior probability and more general dataset shift. The usefulness of this unbiasedness concept is demonstrated with three examples of classifiers used for quantification: Adjusted Classify & Count, EM-algorithm and CDE-Iterate. We find that Adjusted Classify & Count and EM-algorithm are Fisher consistent. A counter-example shows that CDE-Iterate is not Fisher consistent and, therefore, cannot be trusted to deliver reliable estimates of class probabilities.