An AI System Evaluation Framework for Advancing AI Safety: Terminology, Taxonomy, Lifecycle Mapping
This work addresses the need for standardized evaluation methods to improve AI safety, though it is incremental as it builds on existing concepts without introducing new technical breakthroughs.
The paper tackles the problem of divergent practices and terminologies hindering AI safety evaluations across communities by proposing a framework with harmonized terminology, a taxonomy of evaluation elements, and lifecycle mapping to facilitate communication and comprehensive evaluation.
The advent of advanced AI underscores the urgent need for comprehensive safety evaluations, necessitating collaboration across communities (i.e., AI, software engineering, and governance). However, divergent practices and terminologies across these communities, combined with the complexity of AI systems-of which models are only a part-and environmental affordances (e.g., access to tools), obstruct effective communication and comprehensive evaluation. This paper proposes a framework for AI system evaluation comprising three components: 1) harmonised terminology to facilitate communication across communities involved in AI safety evaluation; 2) a taxonomy identifying essential elements for AI system evaluation; 3) a mapping between AI lifecycle, stakeholders, and requisite evaluations for accountable AI supply chain. This framework catalyses a deeper discourse on AI system evaluation beyond model-centric approaches.