From Feature Importance to Natural Language Explanations Using LLMs with RAG
This work addresses the need for conversational explanations in autonomous decision-making involving human interaction, representing an incremental improvement by integrating existing methods like RAG and feature importance into a novel framework.
The paper tackled the problem of making machine learning model outputs understandable through natural language explanations by introducing traceable question-answering using LLMs with RAG, and the result showed that explanations generated by LLMs incorporated key elements from social science to bridge the gap between complex models and human understanding.
As machine learning becomes increasingly integral to autonomous decision-making processes involving human interaction, the necessity of comprehending the model's outputs through conversational means increases. Most recently, foundation models are being explored for their potential as post hoc explainers, providing a pathway to elucidate the decision-making mechanisms of predictive models. In this work, we introduce traceable question-answering, leveraging an external knowledge repository to inform the responses of Large Language Models (LLMs) to user queries within a scene understanding task. This knowledge repository comprises contextual details regarding the model's output, containing high-level features, feature importance, and alternative probabilities. We employ subtractive counterfactual reasoning to compute feature importance, a method that entails analysing output variations resulting from decomposing semantic features. Furthermore, to maintain a seamless conversational flow, we integrate four key characteristics - social, causal, selective, and contrastive - drawn from social science research on human explanations into a single-shot prompt, guiding the response generation process. Our evaluation demonstrates that explanations generated by the LLMs encompassed these elements, indicating its potential to bridge the gap between complex model outputs and natural language expressions.