Conformal Off-Policy Prediction in Contextual Bandits
This work addresses the need for robust policy evaluation in safety-critical applications, offering a novel approach to quantify uncertainty in off-policy predictions.
The paper tackles the problem of evaluating policies in contextual bandits by providing reliable predictive intervals for outcomes, rather than just expected values, to capture variability and ensure safety. It introduces Conformal Off-Policy Prediction (COPP), which offers finite-sample guarantees and shows utility on synthetic and real-world data.
Most off-policy evaluation methods for contextual bandits have focused on the expected outcome of a policy, which is estimated via methods that at best provide only asymptotic guarantees. However, in many applications, the expectation may not be the best measure of performance as it does not capture the variability of the outcome. In addition, particularly in safety-critical settings, stronger guarantees than asymptotic correctness may be required. To address these limitations, we consider a novel application of conformal prediction to contextual bandits. Given data collected under a behavioral policy, we propose \emph{conformal off-policy prediction} (COPP), which can output reliable predictive intervals for the outcome under a new target policy. We provide theoretical finite-sample guarantees without making any additional assumptions beyond the standard contextual bandit setup, and empirically demonstrate the utility of COPP compared with existing methods on synthetic and real-world data.