Programmatic Policy Extraction by Iterative Local Search
This addresses the need for more interpretable and verifiable policies in reinforcement learning, though it is incremental as it builds on existing imitation and local search techniques.
The paper tackles the challenge of extracting interpretable programmatic policies from pretrained neural policies in reinforcement learning, achieving performance nearly as good as the original policies in a pendulum swing-up task.
Reinforcement learning policies are often represented by neural networks, but programmatic policies are preferred in some cases because they are more interpretable, amenable to formal verification, or generalize better. While efficient algorithms for learning neural policies exist, learning programmatic policies is challenging. Combining imitation-projection and dataset aggregation with a local search heuristic, we present a simple and direct approach to extracting a programmatic policy from a pretrained neural policy. After examining our local search heuristic on a programming by example problem, we demonstrate our programmatic policy extraction method on a pendulum swing-up problem. Both when trained using a hand crafted expert policy and a learned neural policy, our method discovers simple and interpretable policies that perform almost as well as the original.