Epistemic Errors of Imperfect Multitask Learners When Distributions Shift
This work addresses the challenge of improving reliability in machine learning models for applications where data distributions shift or tasks are imperfectly learned, though it is incremental in extending existing uncertainty quantification frameworks.
The paper tackles the problem of quantifying and reducing epistemic errors in uncertainty-aware learners under distribution shift and multitask learning, introducing a decompositional error bound that attributes errors to specific aspects of the learning process. It provides specialized bounds for Bayesian transfer learning and distribution shift, and a new definition of negative transfer.
Uncertainty-aware machine learners, such as Bayesian neural networks, output a quantification of uncertainty instead of a point prediction. In this work, we provide uncertainty-aware learners with a principled framework to characterize, and identify ways to eliminate, errors that arise from reducible (epistemic) uncertainty. We introduce a principled definition of epistemic error, and provide a decompositional epistemic error bound which operates in the very general setting of imperfect multitask learning under distribution shift. In this setting, the training (source) data may arise from multiple tasks, the test (target) data may differ systematically from the source data tasks, and/or the learner may not arrive at an accurate characterization of the source data. Our bound separately attributes epistemic errors to each of multiple aspects of the learning procedure and environment. As corollaries of the general result, we provide epistemic error bounds specialized to the settings of Bayesian transfer learning and distribution shift within $ε$-neighborhoods. We additionally leverage the terms in our bound to provide a novel definition of negative transfer.