A Connection Between Learning to Reject and Bhattacharyya Divergences
This work addresses the challenge of improving model reliability in machine learning by providing a novel theoretical connection, though it appears incremental as it builds on existing rejection paradigms.
The paper tackles the problem of learning when to abstain from predictions by linking rejection to thresholding statistical divergences, finding that using a Bhattacharyya divergence leads to less aggressive rejection compared to the standard Chow's Rule.
Learning to reject provide a learning paradigm which allows for our models to abstain from making predictions. One way to learn the rejector is to learn an ideal marginal distribution (w.r.t. the input domain) - which characterizes a hypothetical best marginal distribution - and compares it to the true marginal distribution via a density ratio. In this paper, we consider learning a joint ideal distribution over both inputs and labels; and develop a link between rejection and thresholding different statistical divergences. We further find that when one considers a variant of the log-loss, the rejector obtained by considering the joint ideal distribution corresponds to the thresholding of the skewed Bhattacharyya divergence between class-probabilities. This is in contrast to the marginal case - that is equivalent to a typical characterization of optimal rejection, Chow's Rule - which corresponds to a thresholding of the Kullback-Leibler divergence. In general, we find that rejecting via a Bhattacharyya divergence is less aggressive than Chow's Rule.