Toward Idealized Decision Theory
It addresses the foundational problem of AI alignment for humanity, but is incremental as it builds on recent insights without presenting new methods or results.
The paper identifies shortcomings in standard decision theory formulations for aligning smarter-than-human AI with human interests, and explores policy selection and logical counterfactuals as promising research directions.
This paper motivates the study of decision theory as necessary for aligning smarter-than-human artificial systems with human interests. We discuss the shortcomings of two standard formulations of decision theory, and demonstrate that they cannot be used to describe an idealized decision procedure suitable for approximation by artificial systems. We then explore the notions of policy selection and logical counterfactuals, two recent insights into decision theory that point the way toward promising paths for future research.