Joint Estimation of Expertise and Reward Preferences From Human Demonstrations
This addresses the challenge of adapting robot learning to non-optimal human partners in human-robot interaction, though it is incremental as it builds on existing inference methods.
The paper tackles the problem of robots learning from non-expert human demonstrations by jointly inferring both the human's expertise level and objective function, enabling the robot to adjust assistance and improve skill learning from novice users, with demonstrations in simulation and real user data.
When a robot learns from human examples, most approaches assume that the human partner provides examples of optimal behavior. However, there are applications in which the robot learns from non-expert humans. We argue that the robot should learn not only about the human's objectives, but also about their expertise level. The robot could then leverage this joint information to reduce or increase the frequency at which it provides assistance to its human's partner or be more cautious when learning new skills from novice users. Similarly, by taking into account the human's expertise, the robot would also be able of inferring a human's true objectives even when the human's fails to properly demonstrate these objectives due to a lack of expertise. In this paper, we propose to jointly infer the expertise level and objective function of a human given observations of their (possibly) non-optimal demonstrations. Two inference approaches are proposed. In the first approach, inference is done over a finite, discrete set of possible objective functions and expertise levels. In the second approach, the robot optimizes over the space of all possible hypotheses and finds the objective function and expertise level that best explain the observed human behavior. We demonstrate our proposed approaches both in simulation and with real user data.