Balanced Policy Evaluation and Learning
This work addresses high-variance issues in policy evaluation and learning for applications like personalized medicine and internet advertising, offering a more effective solution compared to prior methods.
The authors tackled the problem of evaluating and learning personalized decision policies from observational data, where only outcomes of enacted decisions are available and the historical policy is unknown, by proposing a balance-based approach that directly optimizes weights to align data with the target policy. This method significantly outperforms existing inverse propensity weighting techniques in both evaluation and learning, as demonstrated through theoretical guarantees and empirical results.
We present a new approach to the problems of evaluating and learning personalized decision policies from observational data of past contexts, decisions, and outcomes. Only the outcome of the enacted decision is available and the historical policy is unknown. These problems arise in personalized medicine using electronic health records and in internet advertising. Existing approaches use inverse propensity weighting (or, doubly robust versions) to make historical outcome (or, residual) data look like it were generated by a new policy being evaluated or learned. But this relies on a plug-in approach that rejects data points with a decision that disagrees with the new policy, leading to high variance estimates and ineffective learning. We propose a new, balance-based approach that too makes the data look like the new policy but does so directly by finding weights that optimize for balance between the weighted data and the target policy in the given, finite sample, which is equivalent to minimizing worst-case or posterior conditional mean square error. Our policy learner proceeds as a two-level optimization problem over policies and weights. We demonstrate that this approach markedly outperforms existing ones both in evaluation and learning, which is unsurprising given the wider support of balance-based weights. We establish extensive theoretical consistency guarantees and regret bounds that support this empirical success.