Context-Action Embedding Learning for Off-Policy Evaluation in Contextual Bandits
This work addresses variance reduction in off-policy evaluation for contextual bandits, an incremental improvement over existing methods like MIPS.
The paper tackles the high variance problem in off-policy evaluation for contextual bandits by introducing CAEL-MIPS, which learns context-action embeddings to minimize mean squared error, and demonstrates improved MSE performance on synthetic and real-world datasets.
We consider off-policy evaluation (OPE) in contextual bandits with finite action space. Inverse Propensity Score (IPS) weighting is a widely used method for OPE due to its unbiased, but it suffers from significant variance when the action space is large or when some parts of the context-action space are underexplored. Recently introduced Marginalized IPS (MIPS) estimators mitigate this issue by leveraging action embeddings. However, these embeddings do not minimize the mean squared error (MSE) of the estimators and do not consider context information. To address these limitations, we introduce Context-Action Embedding Learning for MIPS, or CAEL-MIPS, which learns context-action embeddings from offline data to minimize the MSE of the MIPS estimator. Building on the theoretical analysis of bias and variance of MIPS, we present an MSE-minimizing objective for CAEL-MIPS. In the empirical studies on a synthetic dataset and a real-world dataset, we demonstrate that our estimator outperforms baselines in terms of MSE.