Adaptive Querying for Reward Learning from Human Feedback
This work addresses the challenge of efficiently learning from human feedback in robotics, though it is incremental by building on existing methods with adaptive querying.
The paper tackles the problem of learning penalty functions for unsafe robot behaviors by optimizing both the query state and feedback format, resulting in improved sample efficiency in simulation.
Learning from human feedback is a popular approach to train robots to adapt to user preferences and improve safety. Existing approaches typically consider a single querying (interaction) format when seeking human feedback and do not leverage multiple modes of user interaction with a robot. We examine how to learn a penalty function associated with unsafe behaviors, such as side effects, using multiple forms of human feedback, by optimizing the query state and feedback format. Our framework for adaptive feedback selection enables querying for feedback in critical states in the most informative format, while accounting for the cost and probability of receiving feedback in a certain format. We employ an iterative, two-phase approach which first selects critical states for querying, and then uses information gain to select a feedback format for querying across the sampled critical states. Our evaluation in simulation demonstrates the sample efficiency of our approach.