Causal Analysis of Agent Behavior for AI Safety
This work addresses AI safety by providing tools for human-understandable explanations of agent behavior, though it appears incremental as it builds on existing causal analysis concepts.
The paper tackles the problem of understanding unpredictable and opaque machine learning systems by proposing a methodology for investigating causal mechanisms driving artificial agent behavior, demonstrating that six typical analyst questions require experimental manipulations rather than pure observation.
As machine learning systems become more powerful they also become increasingly unpredictable and opaque. Yet, finding human-understandable explanations of how they work is essential for their safe deployment. This technical report illustrates a methodology for investigating the causal mechanisms that drive the behaviour of artificial agents. Six use cases are covered, each addressing a typical question an analyst might ask about an agent. In particular, we show that each question cannot be addressed by pure observation alone, but instead requires conducting experiments with systematically chosen manipulations so as to generate the correct causal evidence.