Short-Long Policy Evaluation with Novel Actions
This work addresses a bottleneck in innovation cycles for domains like education, healthcare, and consumer technology by enabling faster evaluation of new strategies, though it appears incremental as it builds on prior policy evaluation methods.
The paper tackles the problem of evaluating long-term outcomes of new decision policies without requiring lengthy observations, by introducing a short-long policy evaluation setting for sequential decision making. The proposed methods significantly outperform prior results on simulators of HIV treatment, kidney dialysis, and battery charging, and are shown to be useful for AI safety applications in quickly identifying low-performing policies.
From incorporating LLMs in education, to identifying new drugs and improving ways to charge batteries, innovators constantly try new strategies in search of better long-term outcomes for students, patients and consumers. One major bottleneck in this innovation cycle is the amount of time it takes to observe the downstream effects of a decision policy that incorporates new interventions. The key question is whether we can quickly evaluate long-term outcomes of a new decision policy without making long-term observations. Organizations often have access to prior data about past decision policies and their outcomes, evaluated over the full horizon of interest. Motivated by this, we introduce a new setting for short-long policy evaluation for sequential decision making tasks. Our proposed methods significantly outperform prior results on simulators of HIV treatment, kidney dialysis and battery charging. We also demonstrate that our methods can be useful for applications in AI safety by quickly identifying when a new decision policy is likely to have substantially lower performance than past policies.