Servant of Many Masters: Shifting priorities in Pareto-optimal sequential decision-making
This addresses a foundational issue in multi-agent systems and contract design, with implications for fields like robotics and joint ventures, representing a novel extension rather than an incremental improvement.
The paper tackles the problem of Pareto-optimal sequential decision-making for agents serving multiple principals with different priors, showing that Harsanyi's theorem fails in this case and deriving a generalization where weights on principals' utilities evolve over time based on observational conformity.
It is often argued that an agent making decisions on behalf of two or more principals who have different utility functions should adopt a {\em Pareto-optimal} policy, i.e., a policy that cannot be improved upon for one agent without making sacrifices for another. A famous theorem of Harsanyi shows that, when the principals have a common prior on the outcome distributions of all policies, a Pareto-optimal policy for the agent is one that maximizes a fixed, weighted linear combination of the principals' utilities. In this paper, we show that Harsanyi's theorem does not hold for principals with different priors, and derive a more precise generalization which does hold, which constitutes our main result. In this more general case, the relative weight given to each principal's utility should evolve over time according to how well the agent's observations conform with that principal's prior. The result has implications for the design of contracts, treaties, joint ventures, and robots.