Data-Adaptive Tradeoffs among Multiple Risks in Distribution-Free Prediction
This addresses risk management in uncertainty quantification for practitioners, though it appears incremental as it builds on existing methods to fix violations.
The paper tackles the problem of data-adaptive tradeoffs among multiple risks in decision-making pipelines, showing that naive methods can violate risk guarantees, and develops a distribution-free method to ensure valid control, with experiments on synthetic data and MS-COCO.
Decision-making pipelines are generally characterized by tradeoffs among various risk functions. It is often desirable to manage such tradeoffs in a data-adaptive manner. As we demonstrate, if this is done naively, state-of-the art uncertainty quantification methods can lead to significant violations of putative risk guarantees. To address this issue, we develop methods that permit valid control of risk when threshold and tradeoff parameters are chosen adaptively. Our methodology supports monotone and nearly-monotone risks, but otherwise makes no distributional assumptions. To illustrate the benefits of our approach, we carry out numerical experiments on synthetic data and the large-scale vision dataset MS-COCO.