An Objective Laboratory Protocol for Evaluating Cognition of Non-Human Systems Against Human Cognition
This addresses the need for a comprehensive evaluation method to assess AI safety and ethics, though it appears incremental compared to past attempts like the Turing Test.
The paper tackles the problem of evaluating non-human systems' cognitive capabilities against human cognition by proposing an objective laboratory protocol, which outputs a confidence statistic and identifies specific areas of shortfall.
In this paper I describe and reduce to practice an objective protocol for evaluating the cognitive capabilities of a non-human system against human cognition in a laboratory environment. This is important because the existence of a non-human system with cognitive capabilities comparable to those of humans might make once-philosophical questions of safety and ethics immediate and urgent. Past attempts to devise evaluation methods, such as the Turing Test and many others, have not met this need; most of them either emphasize a single aspect of human cognition or a single theory of intelligence, fail to capture the human capacity for generality and novelty, or require success in the physical world. The protocol is broadly Bayesian, in that its primary output is a confidence statistic in relation to a claim. Further, it provides insight into the areas where and to what extent a particular system falls short of human cognition, which can help to drive further progress or precautions.