MLLM-Microscope: Unlocking Hidden Structure Within Multimodal Large Language Models
For researchers studying multimodal LLM internals, this work offers a new analysis tool and empirical findings, but the insights are incremental and domain-specific.
MLLM-Microscope analyzes hidden representations in Multimodal Large Language Models, revealing that token embeddings exhibit highly linear behaviors across layers, with modality fusion method affecting linearity, intrinsic dimension, and anisotropy. The system provides insights to inform future model design.
This work presents MLLM-Microscope, a novel system designed for analyzing the hidden representations within Multimodal Large Language Models (MLLMs). Our system evaluates the linearity, intrinsic dimension, and anisotropy of multimodal token embeddings across transformer layers. Utilizing the ScienceQA dataset, we evaluate two state-of-the-art MLLMs, LLaVA-NeXT and OmniFusion. We find that both the main and residual streams for tokens of both modalities exhibit highly linear behaviors across transformer layers. However, LLaVA-NeXT's image tokens reveal a slight decline in linearity, whereas OmniFusion's remain consistent. Image token dimensions in OmniFusion remain consistently higher across layers compared to LLaVA-NeXT. Also, the OmniFusion's anisotropy is observed to stay consistently low throughout the layers. These findings suggest that the inner workings of MLLMs highly depend on the nature of modality fusion performed before passing the token sequence into LLM. This and other new potential insights obtainable from our system are surely capable of enhancing our understanding of the inner workings of MLLMs, informing future model design and optimization.