David J. Jilk

David J. Jilk

In this paper I describe and reduce to practice an objective protocol for evaluating the cognitive capabilities of a non-human system against human cognition in a laboratory environment. This is important because the existence of a non-human system with cognitive capabilities comparable to those of humans might make once-philosophical questions of safety and ethics immediate and urgent. Past attempts to devise evaluation methods, such as the Turing Test and many others, have not met this need; most of them either emphasize a single aspect of human cognition or a single theory of intelligence, fail to capture the human capacity for generality and novelty, or require success in the physical world. The protocol is broadly Bayesian, in that its primary output is a confidence statistic in relation to a claim. Further, it provides insight into the areas where and to what extent a particular system falls short of human cognition, which can help to drive further progress or precautions.

David J. Jilk

Verification and validation of agentic behavior have been suggested as important research priorities in efforts to reduce risks associated with the creation of general artificial intelligence (Russell et al 2015). In this paper we question the appropriateness of using language of certainty with respect to efforts to manage that risk. We begin by establishing a very general formalism to characterize agentic behavior and to describe standards of acceptable behavior. We show that determination of whether an agent meets any particular standard is not computable. We discuss the extent of the burden associated with verification by manual proof and by automated behavioral governance. We show that to ensure decidability of the behavioral standard itself, one must further limit the capabilities of the agent. We then demonstrate that if our concerns relate to outcomes in the physical world, attempts at validation are futile. Finally, we show that layered architectures aimed at making these challenges tractable mistakenly equate intentions with actions or outcomes, thereby failing to provide any guarantees. We conclude with a discussion of why language of certainty should be eradicated from the conversation about the safety of general artificial intelligence.