Consistent Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards
This addresses the challenge of designing AI systems that serve multiple stakeholders with varying time preferences, offering a normative solution to a foundational issue in multi-objective agency.
The paper tackles the problem of aggregating multiple objectives with different time preferences in AI agents, showing that Markovian aggregation is impossible under certain axioms and proposing a practical non-Markovian scheme with one additional parameter per objective.
As the capabilities of artificial agents improve, they are being increasingly deployed to service multiple diverse objectives and stakeholders. However, the composition of these objectives is often performed ad hoc, with no clear justification. This paper takes a normative approach to multi-objective agency: from a set of intuitively appealing axioms, it is shown that Markovian aggregation of Markovian reward functions is not possible when the time preference (discount factor) for each objective may vary. It follows that optimal multi-objective agents must admit rewards that are non-Markovian with respect to the individual objectives. To this end, a practical non-Markovian aggregation scheme is proposed, which overcomes the impossibility with only one additional parameter for each objective. This work offers new insights into sequential, multi-objective agency and intertemporal choice, and has practical implications for the design of AI systems deployed to serve multiple generations of principals with varying time preference.