Learning the Preferences of Ignorant, Inconsistent Agents
This work addresses the challenge of accurately learning preferences for applications like personalized recommendations, but it is incremental as it builds on existing Bayesian inverse planning methods by incorporating known human biases.
The paper tackled the problem of inferring human preferences from observed choices by developing generative models that account for deviations from optimal behavior, such as false beliefs and time inconsistency, and demonstrated through a behavioral experiment that both the model and human subjects explain choices by considering these systematic deviations.
An important use of machine learning is to learn what people value. What posts or photos should a user be shown? Which jobs or activities would a person find rewarding? In each case, observations of people's past choices can inform our inferences about their likes and preferences. If we assume that choices are approximately optimal according to some utility function, we can treat preference inference as Bayesian inverse planning. That is, given a prior on utility functions and some observed choices, we invert an optimal decision-making process to infer a posterior distribution on utility functions. However, people often deviate from approximate optimality. They have false beliefs, their planning is sub-optimal, and their choices may be temporally inconsistent due to hyperbolic discounting and other biases. We demonstrate how to incorporate these deviations into algorithms for preference inference by constructing generative models of planning for agents who are subject to false beliefs and time inconsistency. We explore the inferences these models make about preferences, beliefs, and biases. We present a behavioral experiment in which human subjects perform preference inference given the same observations of choices as our model. Results show that human subjects (like our model) explain choices in terms of systematic deviations from optimal behavior and suggest that they take such deviations into account when inferring preferences.