Inferring Rewards from Language in Context
This addresses the challenge of enabling AI systems to understand and act on user preferences in new contexts, though it is incremental as it builds on existing instruction following and inverse reinforcement learning approaches.
The paper tackled the problem of inferring a user's underlying reward function from language in context, rather than just mapping language to actions, and presented a pragmatic model that more accurately infers rewards and predicts optimal actions in unseen environments on a flight-booking task.
In classic instruction following, language like "I'd like the JetBlue flight" maps to actions (e.g., selecting that flight). However, language also conveys information about a user's underlying reward function (e.g., a general preference for JetBlue), which can allow a model to carry out desirable actions in new contexts. We present a model that infers rewards from language pragmatically: reasoning about how speakers choose utterances not only to elicit desired actions, but also to reveal information about their preferences. On a new interactive flight-booking task with natural language, our model more accurately infers rewards and predicts optimal actions in unseen environments, in comparison to past work that first maps language to actions (instruction following) and then maps actions to rewards (inverse reinforcement learning).