Axiomatic Choice and the Decision-Evaluation Paradox
This work addresses foundational issues in decision-making for AI and ethics, but it appears incremental as it builds on existing axiomatic frameworks without introducing a new paradigm.
The paper tackles the problem of modeling decisions with axioms, such as ethical constraints, and identifies a tension between using axioms to make decisions and to evaluate decisions, termed the Decision-Evaluation Paradox, which highlights the need for caution in training models on decision data.
We introduce a framework for modeling decisions with axioms that are statements about decisions, e.g., ethical constraints. Using our framework we define a taxonomy of decision axioms based on their structural properties and demonstrate a tension between the use of axioms to make decisions and the use of axioms to evaluate decisions which we call the Decision-Evaluation Paradox. We argue that the Decision-Evaluation Paradox arises with realistic axiom structures, and the paradox illuminates why one must be exceptionally careful when training models on decision data or applying axioms to make and evaluate decisions.