Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filtering
This work addresses the need for explainability in MLLMs for researchers and practitioners, though it is incremental as it builds on existing neuron identification techniques.
The study tackled the problem of opaque internal processing and hallucinations in multimodal large language models (MLLMs) by proposing a two-stage filtering method to identify knowledge neurons in MiniGPT-4, achieving higher accuracy in locating knowledge compared to existing methods on the MS COCO 2017 dataset with BLEU, ROUGE, and BERTScore evaluations.
Recent advances in large language models (LLMs) have led to the development of multimodal LLMs (MLLMs) in the fields of natural language processing (NLP) and computer vision. Although these models allow for integrated visual and language understanding, they present challenges such as opaque internal processing and the generation of hallucinations and misinformation. Therefore, there is a need for a method to clarify the location of knowledge in MLLMs. In this study, we propose a method to identify neurons associated with specific knowledge using MiniGPT-4, a Transformer-based MLLM. Specifically, we extract knowledge neurons through two stages: activation differences filtering using inpainting and gradient-based filtering using GradCAM. Experiments on the image caption generation task using the MS COCO 2017 dataset, BLEU, ROUGE, and BERTScore quantitative evaluation, and qualitative evaluation using an activation heatmap showed that our method is able to locate knowledge with higher accuracy than existing methods. This study contributes to the visualization and explainability of knowledge in MLLMs and shows the potential for future knowledge editing and control.