Statistical Scenario Modelling and Lookalike Distributions for Multi-Variate AI Risk
This work addresses the lack of statistically grounded methods for assessing aggregated AI risk in modular workflows, which is an incremental contribution to AI safety research.
The paper tackles the problem of evaluating AI safety by developing statistical methods to model AI risk holistically and estimate AI impacts using analogous data, demonstrating their utility through risk analysis of logistic scenario simulations.
Evaluating AI safety requires statistically rigorous methods and risk metrics for understanding how the use of AI affects aggregated risk. However, much AI safety literature focuses upon risks arising from AI models in isolation, lacking consideration of how modular use of AI affects risk distribution of workflow components or overall risk metrics. There is also a lack of statistical grounding enabling sensitisation of risk models in the presence of absence of AI to estimate causal contributions of AI. This is in part due to the dearth of AI impact data upon which to fit distributions. In this work, we address these gaps in two ways. First, we demonstrate how scenario modelling (grounded in established statistical techniques such as Markov chains, copulas and Monte Carlo simulation) can be used to model AI risk holistically. Second, we show how lookalike distributions from phenomena analogous to AI can be used to estimate AI impacts in the absence of directly observable data. We demonstrate the utility of our methods for benchmarking cumulative AI risk via risk analysis of a logistic scenario simulations.