Using Thought-Provoking Children's Questions to Drive Artificial Intelligence Research
This work addresses the challenge of assessing AI reasoning capabilities for researchers, but it is incremental as it introduces a new evaluation task without demonstrating AI performance.
The authors tackled the problem of evaluating general-purpose AI systems by proposing thought-provoking children's questions (TPCQs) as a new method, analyzing 244 BrainPlay questions to show they span many aspects of intelligence and suggesting human judges for evaluation due to open-ended answers.
We propose to use thought-provoking children's questions (TPCQs), namely Highlights BrainPlay questions, as a new method to drive artificial intelligence research and to evaluate the capabilities of general-purpose AI systems. These questions are designed to stimulate thought and learning in children, and they can be used to do the same thing in AI systems, while demonstrating the system's reasoning capabilities to the evaluator. We introduce the TPCQ task, which which takes a TPCQ question as input and produces as output (1) answers to the question and (2) learned generalizations. We discuss how BrainPlay questions stimulate learning. We analyze 244 BrainPlay questions, and we report statistics on question type, question class, answer cardinality, answer class, types of knowledge needed, and types of reasoning needed. We find that BrainPlay questions span many aspects of intelligence. Because the answers to BrainPlay questions and the generalizations learned from them are often highly open-ended, we suggest using human judges for evaluation.