Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift
This work addresses uncertainty quantification for machine learning models in dynamic environments with distribution shifts, providing a method for practitioners to improve reliability, though it is incremental as it builds on existing prediction interval aggregation techniques.
The paper tackles the problem of aggregating prediction intervals to achieve minimal width and adequate coverage under unsupervised domain shift, where labeled source data and unlabeled target covariates are available, and demonstrates performance with real-world datasets and theoretical guarantees.
As machine learning models are increasingly deployed in dynamic environments, it becomes paramount to assess and quantify uncertainties associated with distribution shifts. A distribution shift occurs when the underlying data-generating process changes, leading to a deviation in the model's performance. The prediction interval, which captures the range of likely outcomes for a given prediction, serves as a crucial tool for characterizing uncertainties induced by their underlying distribution. In this paper, we propose methodologies for aggregating prediction intervals to obtain one with minimal width and adequate coverage on the target domain under unsupervised domain shift, under which we have labeled samples from a related source domain and unlabeled covariates from the target domain. Our analysis encompasses scenarios where the source and the target domain are related via i) a bounded density ratio, and ii) a measure-preserving transformation. Our proposed methodologies are computationally efficient and easy to implement. Beyond illustrating the performance of our method through real-world datasets, we also delve into the theoretical details. This includes establishing rigorous theoretical guarantees, coupled with finite sample bounds, regarding the coverage and width of our prediction intervals. Our approach excels in practical applications and is underpinned by a solid theoretical framework, ensuring its reliability and effectiveness across diverse contexts.