DOLCE: Decomposing Off-Policy Evaluation/Learning into Lagged and Current Effects
This addresses a key limitation in contextual bandit policies for settings requiring explicit evaluation or optimization for individuals outside common support, representing a novel method for a known bottleneck.
The paper tackles the problem of off-policy evaluation and learning when the common support assumption is violated, proposing DOLCE, which decomposes rewards into lagged and current effects using contextual information from multiple time points, and shows substantial improvements in experiments as the proportion of individuals outside common support increases.
Off-policy evaluation (OPE) and off-policy learning (OPL) for contextual bandit policies leverage historical data to evaluate and optimize a target policy. Most existing OPE/OPL methods--based on importance weighting or imputation--assume common support between the target and logging policies. When this assumption is violated, these methods typically require unstable extrapolation, truncation, or conservative strategies for individuals outside the common support assumption. However, such approaches can be inadequate in settings where explicit evaluation or optimization for such individuals is required. To address this issue, we propose DOLCE: Decomposing Off-policy evaluation/learning into Lagged and Current Effects, a novel estimator that leverages contextual information from multiple time points to decompose rewards into lagged and current effects. By incorporating both past and present contexts, DOLCE effectively handles individuals who violate the common support assumption. We show that the proposed estimator is unbiased under two assumptions--local correctness and conditional independence. Our experiments demonstrate that DOLCE achieves substantial improvements in OPE and OPL, particularly as the proportion of individuals outside the common support assumption increases.