TokenSHAP: Interpreting Large Language Models with Monte Carlo Shapley Value Estimation
This addresses the need for interpretable AI in critical applications by providing a rigorous framework for understanding LLM behavior, enhancing transparency and reliability, though it is an incremental adaptation of existing methods to a new domain.
The authors tackled the problem of interpreting large language models (LLMs) by introducing TokenSHAP, a method that attributes importance to tokens in input prompts using Shapley values and Monte Carlo sampling, resulting in consistent improvements over baselines in alignment with human judgments, faithfulness, and consistency.
As large language models (LLMs) become increasingly prevalent in critical applications, the need for interpretable AI has grown. We introduce TokenSHAP, a novel method for interpreting LLMs by attributing importance to individual tokens or substrings within input prompts. This approach adapts Shapley values from cooperative game theory to natural language processing, offering a rigorous framework for understanding how different parts of an input contribute to a model's response. TokenSHAP leverages Monte Carlo sampling for computational efficiency, providing interpretable, quantitative measures of token importance. We demonstrate its efficacy across diverse prompts and LLM architectures, showing consistent improvements over existing baselines in alignment with human judgments, faithfulness to model behavior, and consistency. Our method's ability to capture nuanced interactions between tokens provides valuable insights into LLM behavior, enhancing model transparency, improving prompt engineering, and aiding in the development of more reliable AI systems. TokenSHAP represents a significant step towards the necessary interpretability for responsible AI deployment, contributing to the broader goal of creating more transparent, accountable, and trustworthy AI systems.