Limitations on Safe, Trusted, Artificial General Intelligence
This work addresses foundational limitations in AI safety and AGI theory, showing that achieving both safety and AGI under strict definitions is impossible, which is a theoretical result with implications for AI researchers and developers.
The paper proposes strict mathematical definitions of safety, trust, and AGI, and proves a fundamental incompatibility: a safe and trusted AI system cannot be an AGI system, as there exist tasks solvable by humans but not by such a system, demonstrated in domains like program verification and planning.
Safety, trust and Artificial General Intelligence (AGI) are aspirational goals in artificial intelligence (AI) systems, and there are several informal interpretations of these notions. In this paper, we propose strict, mathematical definitions of safety, trust, and AGI, and demonstrate a fundamental incompatibility between them. We define safety of a system as the property that it never makes any false claims, trust as the assumption that the system is safe, and AGI as the property of an AI system always matching or exceeding human capability. Our core finding is that -- for our formal definitions of these notions -- a safe and trusted AI system cannot be an AGI system: for such a safe, trusted system there are task instances which are easily and provably solvable by a human but not by the system. We note that we consider strict mathematical definitions of safety and trust, and it is possible for real-world deployments to instead rely on alternate, practical interpretations of these notions. We show our results for program verification, planning, and graph reachability. Our proofs draw parallels to Gödel's incompleteness theorems and Turing's proof of the undecidability of the halting problem, and can be regarded as interpretations of Gödel's and Turing's results.