Interpretable Preference-based Reinforcement Learning with Tree-Structured Reward Functions
This addresses the reward engineering bottleneck for RL developers by providing interpretable reward structures, though it is incremental as it builds on existing PbRL methods.
The paper tackles the interpretability problem in preference-based reinforcement learning by proposing an online algorithm that learns tree-structured reward functions from human feedback, achieving sample-efficient learning in multiple environments and enabling alignment debugging.
The potential of reinforcement learning (RL) to deliver aligned and performant agents is partially bottlenecked by the reward engineering problem. One alternative to heuristic trial-and-error is preference-based RL (PbRL), where a reward function is inferred from sparse human feedback. However, prior PbRL methods lack interpretability of the learned reward structure, which hampers the ability to assess robustness and alignment. We propose an online, active preference learning algorithm that constructs reward functions with the intrinsically interpretable, compositional structure of a tree. Using both synthetic and human-provided feedback, we demonstrate sample-efficient learning of tree-structured reward functions in several environments, then harness the enhanced interpretability to explore and debug for alignment.