Batch Active Learning of Reward Functions from Human Preferences
This work addresses the challenge of expensive data labeling in robot learning by enabling more efficient preference-based learning, though it is incremental as it builds on existing active querying methods.
The paper tackles the problem of efficiently learning reward functions from human preferences in robotics by developing batch active learning algorithms that reduce data needs and query generation time, achieving results with only a few queries computed quickly in simulation tasks.
Data generation and labeling are often expensive in robot learning. Preference-based learning is a concept that enables reliable labeling by querying users with preference questions. Active querying methods are commonly employed in preference-based learning to generate more informative data at the expense of parallelization and computation time. In this paper, we develop a set of novel algorithms, batch active preference-based learning methods, that enable efficient learning of reward functions using as few data samples as possible while still having short query generation times and also retaining parallelizability. We introduce a method based on determinantal point processes (DPP) for active batch generation and several heuristic-based alternatives. Finally, we present our experimental results for a variety of robotics tasks in simulation. Our results suggest that our batch active learning algorithm requires only a few queries that are computed in a short amount of time. We showcase one of our algorithms in a study to learn human users' preferences.