Model-Agnostic Policy Explanations with Large Language Models
This addresses the need for interpretability in AI agents to foster human trust and meet ethical standards, representing an incremental improvement in explanation generation methods.
The paper tackles the problem of explaining black-box agent behavior in human-centric environments by proposing a model-agnostic method that uses a surrogate model and large language models to generate natural language explanations, resulting in more comprehensible and correct explanations as validated by both automated metrics and human evaluators, with participants better predicting future actions.
Intelligent agents, such as robots, are increasingly deployed in real-world, human-centric environments. To foster appropriate human trust and meet legal and ethical standards, these agents must be able to explain their behavior. However, state-of-the-art agents are typically driven by black-box models like deep neural networks, limiting their interpretability. We propose a method for generating natural language explanations of agent behavior based only on observed states and actions -- without access to the agent's underlying model. Our approach learns a locally interpretable surrogate model of the agent's behavior from observations, which then guides a large language model to generate plausible explanations with minimal hallucination. Empirical results show that our method produces explanations that are more comprehensible and correct than those from baselines, as judged by both language models and human evaluators. Furthermore, we find that participants in a user study more accurately predicted the agent's future actions when given our explanations, suggesting improved understanding of agent behavior.