Learning with Selectively Labeled Data from Multiple Decision-makers
This work addresses selection bias in classification for applications with historical decision data from multiple sources, offering a method to improve model robustness, though it is incremental in extending existing instrumental variable frameworks.
The paper tackles classification with selectively labeled data, where historical decisions from multiple decision-makers create selection bias, and proposes a unified cost-sensitive learning approach to learn robust classifiers, achieving exact identification of classification risk under certain conditions and providing tight bounds otherwise.
We study the problem of classification with selectively labeled data, whose distribution may differ from the full population due to historical decision-making. We exploit the fact that in many applications historical decisions were made by multiple decision-makers, each with different decision rules. We analyze this setup under a principled instrumental variable (IV) framework and rigorously study the identification of classification risk. We establish conditions for the exact identification of classification risk and derive tight partial identification bounds when exact identification fails. We further propose a unified cost-sensitive learning (UCL) approach to learn classifiers robust to selection bias in both identification settings. Finally, we theoretically and numerically validate the efficacy of our proposed method.