Grounding the Ungrounded: A Spectral-Graph Framework for Quantifying Hallucinations in Multimodal LLMs
This addresses the reliability issue of hallucinations in multimodal LLMs for AI safety and evaluation, offering a principled approach to measurement and mitigation.
The authors tackled the problem of quantifying hallucinations in multimodal LLMs by developing a spectral-graph framework based on diffusion dynamics, resulting in a semantic-distortion metric that provides measurable and bounded hallucination energy with interpretable measures.
Hallucinations in LLMs--especially in multimodal settings--undermine reliability. We present a rigorous, information-geometric framework in diffusion dynamics that quantifies hallucination in MLLMs: model outputs are embedded spectrally on multimodal graph Laplacians, and gaps to a truth manifold define a semantic-distortion metric. We derive Courant--Fischer bounds on a temperature-dependent hallucination energy and use RKHS eigenmodes to obtain modality-aware, interpretable measures that track evolution over prompts and time. This reframes hallucination as measurable and bounded, providing a principled basis for evaluation and mitigation.