Composing Efficient, Robust Tests for Policy Selection
This addresses the challenge of efficient policy selection for deployment in real-world applications, though it appears incremental as it builds on existing test selection methods with robustness guarantees.
The paper tackles the problem of selecting a small set of test cases for evaluating reinforcement learning policies under many environmental conditions, introducing RPOSST, which achieves provable robustness and identifies high-quality policies in games, poker datasets, and a racing simulator.
Modern reinforcement learning systems produce many high-quality policies throughout the learning process. However, to choose which policy to actually deploy in the real world, they must be tested under an intractable number of environmental conditions. We introduce RPOSST, an algorithm to select a small set of test cases from a larger pool based on a relatively small number of sample evaluations. RPOSST treats the test case selection problem as a two-player game and optimizes a solution with provable $k$-of-$N$ robustness, bounding the error relative to a test that used all the test cases in the pool. Empirical results demonstrate that RPOSST finds a small set of test cases that identify high quality policies in a toy one-shot game, poker datasets, and a high-fidelity racing simulator.