Preparing for the Intelligence Explosion
This addresses the need for preparedness in the face of potential intelligence explosion scenarios, with broad implications for humanity, though it is incremental in building on existing discussions of AI risks.
The paper tackles the problem of preparing for rapid technological progress driven by AI, which could lead to a range of grand challenges such as new weapons, AI-enabled autocracies, and moral dilemmas, and suggests that proactive measures today are necessary to improve future prospects.
AI that can accelerate research could drive a century of technological progress over just a few years. During such a period, new technological or political developments will raise consequential and hard-to-reverse decisions, in rapid succession. We call these developments grand challenges. These challenges include new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, and digital beings worthy of moral consideration, as well as opportunities to dramatically improve quality of life and collective decision-making. We argue that these challenges cannot always be delegated to future AI systems, and suggest things we can do today to meaningfully improve our prospects. AGI preparedness is therefore not just about ensuring that advanced AI systems are aligned: we should be preparing, now, for the disorienting range of developments an intelligence explosion would bring.