I've got the "Answer"! Interpretation of LLMs Hidden States in Question Answering
This work addresses interpretability for AI researchers and practitioners, but it is incremental as it applies existing analysis methods to new models and data.
The paper tackled the problem of interpreting large language models (LLMs) in knowledge-based question answering by hypothesizing that correct and incorrect behaviors can be distinguished at the hidden state level, and the results supported this hypothesis, identifying specific layers with negative effects.
Interpretability and explainability of AI are becoming increasingly important in light of the rapid development of large language models (LLMs). This paper investigates the interpretation of LLMs in the context of the knowledge-based question answering. The main hypothesis of the study is that correct and incorrect model behavior can be distinguished at the level of hidden states. The quantized models LLaMA-2-7B-Chat, Mistral-7B, Vicuna-7B and the MuSeRC question-answering dataset are used to test this hypothesis. The results of the analysis support the proposed hypothesis. We also identify the layers which have a negative effect on the model's behavior. As a prospect of practical application of the hypothesis, we propose to train such "weak" layers additionally in order to improve the quality of the task solution.