Imagining and building wise machines: The centrality of AI metacognition
This addresses the issue of AI systems being unwise and potentially unsafe for users, but it is incremental as it builds on existing concepts of wisdom and metacognition.
The paper tackles the problem of AI lacking wisdom by analyzing human wisdom as strategies for intractable problems, including metacognitive ones, and argues that improving AI metacognition would enhance robustness, explainability, cooperation, and safety.
Although AI has become increasingly smart, its wisdom has not kept pace. In this article, we examine what is known about human wisdom and sketch a vision of its AI counterpart. We analyze human wisdom as a set of strategies for solving intractable problems-those outside the scope of analytic techniques-including both object-level strategies like heuristics [for managing problems] and metacognitive strategies like intellectual humility, perspective-taking, or context-adaptability [for managing object-level strategies]. We argue that AI systems particularly struggle with metacognition; improved metacognition would lead to AI more robust to novel environments, explainable to users, cooperative with others, and safer in risking fewer misaligned goals with human users. We discuss how wise AI might be benchmarked, trained, and implemented.