Learning Multimodal Rewards from Rankings
This work addresses the challenge of learning diverse reward functions in robotics from human feedback, which is incremental as it extends existing unimodal methods to multimodal settings.
The paper tackles the problem of learning multimodal reward functions from human rankings, addressing scenarios where expert feedback is not unimodal, such as with multiple experts or tasks. It introduces a mixture learning approach with active querying, showing improved data-efficiency in experiments on LunarLander and a Fetch robot.
Learning from human feedback has shown to be a useful approach in acquiring robot reward functions. However, expert feedback is often assumed to be drawn from an underlying unimodal reward function. This assumption does not always hold including in settings where multiple experts provide data or when a single expert provides data for different tasks -- we thus go beyond learning a unimodal reward and focus on learning a multimodal reward function. We formulate the multimodal reward learning as a mixture learning problem and develop a novel ranking-based learning approach, where the experts are only required to rank a given set of trajectories. Furthermore, as access to interaction data is often expensive in robotics, we develop an active querying approach to accelerate the learning process. We conduct experiments and user studies using a multi-task variant of OpenAI's LunarLander and a real Fetch robot, where we collect data from multiple users with different preferences. The results suggest that our approach can efficiently learn multimodal reward functions, and improve data-efficiency over benchmark methods that we adapt to our learning problem.