Model evaluation for extreme risks
This work tackles the critical issue of mitigating extreme risks from advanced AI for policymakers and developers, though it is incremental as it builds on existing evaluation concepts.
The paper addresses the problem of AI systems developing harmful capabilities that could pose extreme risks, arguing that model evaluation is essential for identifying dangerous capabilities and alignment to prevent harm.
Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why model evaluation is critical for addressing extreme risks. Developers must be able to identify dangerous capabilities (through "dangerous capability evaluations") and the propensity of models to apply their capabilities for harm (through "alignment evaluations"). These evaluations will become critical for keeping policymakers and other stakeholders informed, and for making responsible decisions about model training, deployment, and security.