Argumentative Reward Learning: Reasoning About Human Preferences
This work addresses the challenge of efficiently and robustly aligning AI systems with human preferences, which is incremental as it builds on existing reinforcement learning from human feedback methods.
The paper tackles the problem of learning reward functions from human feedback by introducing a neuro-symbolic framework that combines preference-based argumentation with reinforcement learning, resulting in improved generalization of human preferences, reduced user burden, and increased robustness of the reward model.
We define a novel neuro-symbolic framework, argumentative reward learning, which combines preference-based argumentation with existing approaches to reinforcement learning from human feedback. Our method improves prior work by generalising human preferences, reducing the burden on the user and increasing the robustness of the reward model. We demonstrate this with a number of experiments.