Learning Interpretable Models of Aircraft Handling Behaviour by Reinforcement Learning from Human Feedback
This work addresses the need for interpretable models in high-stakes domains like aviation, though it appears incremental as it builds on existing RL and preference learning techniques.
The authors tackled the problem of capturing fast jet pilot handling abilities by developing a method that uses reinforcement learning from human preference feedback to learn an interpretable rule-based model called a reward tree, which achieved competitive performance with uninterpretable neural network models in experiments.
We propose a method to capture the handling abilities of fast jet pilots in a software model via reinforcement learning (RL) from human preference feedback. We use pairwise preferences over simulated flight trajectories to learn an interpretable rule-based model called a reward tree, which enables the automated scoring of trajectories alongside an explanatory rationale. We train an RL agent to execute high-quality handling behaviour by using the reward tree as the objective, and thereby generate data for iterative preference collection and further refinement of both tree and agent. Experiments with synthetic preferences show reward trees to be competitive with uninterpretable neural network reward models on quantitative and qualitative evaluations.