Free Energy Risk Metrics for Systemically Safe AI: Gatekeeping Multi-Agent Study
This addresses the problem of systemic safety in multi-agent AI systems for researchers and practitioners, offering a more transparent and flexible alternative to data-heavy approaches.
The paper tackles the problem of measuring risk in multi-agent AI systems by introducing a Cumulative Risk Exposure metric based on the Free Energy Principle, which requires stakeholders to specify only outcome preferences rather than complex world models. In a simplified autonomous vehicle environment, they demonstrate that gatekeepers using this approach can significantly increase system safety even at low penetration rates.
We investigate the Free Energy Principle as a foundation for measuring risk in agentic and multi-agent systems. From these principles we introduce a Cumulative Risk Exposure metric that is flexible to differing contexts and needs. We contrast this to other popular theories for safe AI that hinge on massive amounts of data or describing arbitrarily complex world models. In our framework, stakeholders need only specify their preferences over system outcomes, providing straightforward and transparent decision rules for risk governance and mitigation. This framework naturally accounts for uncertainty in both world model and preference model, allowing for decision-making that is epistemically and axiologically humble, parsimonious, and future-proof. We demonstrate this novel approach in a simplified autonomous vehicle environment with multi-agent vehicles whose driving policies are mediated by gatekeepers that evaluate, in an online fashion, the risk to the collective safety in their neighborhood, and intervene through each vehicle's policy when appropriate. We show that the introduction of gatekeepers in an AV fleet, even at low penetration, can generate significant positive externalities in terms of increased system safety.