Accounting for Context: Shaping Moral Credences for Value Alignment
For AI alignment researchers, it highlights a fundamental limitation of current moral uncertainty aggregation methods when applied to real-world settings.
The paper argues that aggregating moral evaluations across different theories must account for contextual factors (e.g., uncertainty about action outcomes), and shows that ignoring context leads to violations of the weak Pareto principle, which is a form of Simpson's paradox.
Ensuring that agent behaviours are aligned with human moral values inevitably raises the problem of how to account for the plurality of moral perspectives that societies -- and even individuals -- typically adopt. Work on moral uncertainty proposes mechanisms to fairly and democratically aggregate evaluations of actions across different moral theories. However, this paper argues that one needs to account for contextual factors when aggregating moral evaluations. For example, consequentialist perspectives assume an ability to accurately determine how an agent's actions change the world; an assumption that often does not hold in real world settings. We, therefore, formalise agent decision making under moral uncertainty, while also accounting for these kinds of contextual factors. We thereby show that a seemingly commonsensical property -- the weak Pareto principle -- is violated. We argue that this apparent problem is, in fact, a variation of Simpson's paradox, and hence reveals the limitations of aggregation mechanisms that ignore the impact of contextual factors.