Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention
This addresses the problem of opaque decision-making in LLMs for users needing transparency and accountability, representing an incremental advance by integrating multiple explanation dimensions.
The authors tackled the lack of comprehensive interpretability in Large Language Models by proposing SparseCBM, a sparsity-guided framework that provides holistic explanations across input, subnetwork, and concept levels, and includes interpretable inference-time intervention, demonstrating improved understanding and error correction in empirical evaluations.
Large Language Models (LLMs) have achieved unprecedented breakthroughs in various natural language processing domains. However, the enigmatic ``black-box'' nature of LLMs remains a significant challenge for interpretability, hampering transparent and accountable applications. While past approaches, such as attention visualization, pivotal subnetwork extraction, and concept-based analyses, offer some insight, they often focus on either local or global explanations within a single dimension, occasionally falling short in providing comprehensive clarity. In response, we propose a novel methodology anchored in sparsity-guided techniques, aiming to provide a holistic interpretation of LLMs. Our framework, termed SparseCBM, innovatively integrates sparsity to elucidate three intertwined layers of interpretation: input, subnetwork, and concept levels. In addition, the newly introduced dimension of interpretable inference-time intervention facilitates dynamic adjustments to the model during deployment. Through rigorous empirical evaluations on real-world datasets, we demonstrate that SparseCBM delivers a profound understanding of LLM behaviors, setting it apart in both interpreting and ameliorating model inaccuracies. Codes are provided in supplements.