Can LLMs Explain Themselves Counterfactually?
This addresses the challenge of interpretability and trust in AI systems for users and regulators, but it is incremental as it builds on existing self-explanation methods.
The paper tackled the problem of evaluating whether large language models (LLMs) can generate effective self-explanations, specifically counterfactual ones, and found that LLMs often struggle with this task and their predictions may not align with their own reasoning.
Explanations are an important tool for gaining insights into the behavior of ML models, calibrating user trust and ensuring regulatory compliance. Past few years have seen a flurry of post-hoc methods for generating model explanations, many of which involve computing model gradients or solving specially designed optimization problems. However, owing to the remarkable reasoning abilities of Large Language Model (LLMs), self-explanation, that is, prompting the model to explain its outputs has recently emerged as a new paradigm. In this work, we study a specific type of self-explanations, self-generated counterfactual explanations (SCEs). We design tests for measuring the efficacy of LLMs in generating SCEs. Analysis over various LLM families, model sizes, temperature settings, and datasets reveals that LLMs sometimes struggle to generate SCEs. Even when they do, their prediction often does not agree with their own counterfactual reasoning.