Prediction Sets for Counterfactual Decisions: Coverage, Optimality, and Conformal Prediction
For practitioners using predictions to guide high-stakes decisions (e.g., treatment selection, policy making), this work provides a principled way to construct prediction sets that are both statistically valid and decision-optimal, addressing a critical gap in counterfactual settings.
The paper develops a decision-theoretic framework for uncertainty quantification in counterfactual decisions, introducing policy-coupled coverage as the optimal interface between uncertainty and action. It proposes PC-RACP, a conformal prediction method that achieves higher utility than existing approaches while maintaining valid finite-sample coverage, as demonstrated in simulations and a real email-marketing experiment.
Predictions are increasingly used to guide high-stakes decisions, from treatment selection to policy making. To ensure reliability with imperfect predictions, uncertainty quantification methods such as conformal prediction build prediction sets with coverage guarantees. However, statistical validity alone does not immediately determine the decisions to take, nor the optimality thereof. This gap is especially delicate in counterfactual settings where the outcome that materializes depends on the action taken, so uncertainty cannot be specified independently of the decision rule. We develop a decision-theoretic framework for uncertainty-informed counterfactual decisions. We identify a novel notion of \emph{policy-coupled coverage} -- namely, coverage of the realized outcome under the action induced by the prediction sets themselves -- as the optimal and lossless interface between uncertainty and action. It plays three roles. First, it justifies acting via a natural max-min rule as minimax-optimal under distributional ambiguity. Second, optimizing prediction sets under policy-coupled coverage is equivalent both to a stronger universal-coverage formulation and to the direct risk-averse optimization over policies and utility certificates; this equivalence yields the explicit form of the population-optimal prediction sets. Third, it admits a two-stage procedure, Policy-Coupled Risk-Averse Conformal Prediction (PC-RACP), that approximates these optimal sets with rigorous finite-sample coverage. Simulations and a real email-marketing experiment confirm that PC-RACP delivers higher utility than existing approaches while maintaining valid coverage, and that ignoring the counterfactual structure of the decision problem is suboptimal for both validity and utility.