Towards Mixed Optimization for Reinforcement Learning with Program Synthesis
This addresses the need for interpretable and specifiable policies in reinforcement learning, though it is incremental as it builds on existing synthesis and behavior cloning techniques.
The paper tackles the problem of uninterpretable black-box neural network policies in reinforcement learning by introducing MORL, an iterative framework that uses program synthesis to obtain symbolic representations of policies, enabling manual or automatic debugging and repair. They demonstrate this approach on the CartPole problem, showing that programmatic representations allow high-level modifications that improve policy learning.
Deep reinforcement learning has led to several recent breakthroughs, though the learned policies are often based on black-box neural networks. This makes them difficult to interpret and to impose desired specification constraints during learning. We present an iterative framework, MORL, for improving the learned policies using program synthesis. Concretely, we propose to use synthesis techniques to obtain a symbolic representation of the learned policy, which can then be debugged manually or automatically using program repair. After the repair step, we use behavior cloning to obtain the policy corresponding to the repaired program, which is then further improved using gradient descent. This process continues until the learned policy satisfies desired constraints. We instantiate MORL for the simple CartPole problem and show that the programmatic representation allows for high-level modifications that in turn lead to improved learning of the policies.