Deep Reinforcement Learning from Hierarchical Preference Design
This work addresses the problem of inefficient reward design for researchers and practitioners in reinforcement learning, offering a method that is incremental by building on existing preference-based approaches.
The paper tackles the challenge of reward design in reinforcement learning by proposing a hierarchical reward modeling framework (HERON) that exploits hierarchical structures in feedback signals or uses surrogate feedback for sparse rewards, resulting in high-performing agents with improved sample efficiency and robustness across various tasks.
Reward design is a fundamental, yet challenging aspect of reinforcement learning (RL). Researchers typically utilize feedback signals from the environment to handcraft a reward function, but this process is not always effective due to the varying scale and intricate dependencies of the feedback signals. This paper shows by exploiting certain structures, one can ease the reward design process. Specifically, we propose a hierarchical reward modeling framework -- HERON for scenarios: (I) The feedback signals naturally present hierarchy; (II) The reward is sparse, but with less important surrogate feedback to help policy learning. Both scenarios allow us to design a hierarchical decision tree induced by the importance ranking of the feedback signals to compare RL trajectories. With such preference data, we can then train a reward model for policy learning. We apply HERON to several RL applications, and we find that our framework can not only train high performing agents on a variety of difficult tasks, but also provide additional benefits such as improved sample efficiency and robustness. Our code is available at \url{https://github.com/abukharin3/HERON}.