Causal Explanations for Sequential Decision-Making in Multi-Agent Systems
This addresses the need for trustworthy autonomous agents in dynamic multi-agent environments, though it is incremental as it builds on existing causal explanation methods.
The paper tackles the problem of explaining decisions in multi-agent systems by introducing CEMA, a framework that generates causal natural language explanations using counterfactual simulations from probabilistic models. Results show CEMA correctly identifies decision causes in autonomous driving scenarios and improves user trust comparably to high-quality baselines.
We present CEMA: Causal Explanations in Multi-Agent systems; a framework for creating causal natural language explanations of an agent's decisions in dynamic sequential multi-agent systems to build more trustworthy autonomous agents. Unlike prior work that assumes a fixed causal structure, CEMA only requires a probabilistic model for forward-simulating the state of the system. Using such a model, CEMA simulates counterfactual worlds that identify the salient causes behind the agent's decisions. We evaluate CEMA on the task of motion planning for autonomous driving and test it in diverse simulated scenarios. We show that CEMA correctly and robustly identifies the causes behind the agent's decisions, even when a large number of other agents is present, and show via a user study that CEMA's explanations have a positive effect on participants' trust in autonomous vehicles and are rated as high as high-quality baseline explanations elicited from other participants. We release the collected explanations with annotations as the HEADD dataset.