Temporal Insight Enhancement: Mitigating Temporal Hallucination in Multimodal Large Language Models
This addresses a critical limitation in MLLMs for video understanding, though it is an incremental improvement focusing on a specific domain.
The study tackled the problem of event-level hallucinations in Multimodal Large Language Models (MLLMs) when processing video inputs, resulting in a significant reduction in temporal hallucinations and improved event-related response quality as demonstrated on the Charades-STA dataset.
Recent advancements in Multimodal Large Language Models (MLLMs) have significantly enhanced the comprehension of multimedia content, bringing together diverse modalities such as text, images, and videos. However, a critical challenge faced by these models, especially when processing video inputs, is the occurrence of hallucinations - erroneous perceptions or interpretations, particularly at the event level. This study introduces an innovative method to address event-level hallucinations in MLLMs, focusing on specific temporal understanding in video content. Our approach leverages a novel framework that extracts and utilizes event-specific information from both the event query and the provided video to refine MLLMs' response. We propose a unique mechanism that decomposes on-demand event queries into iconic actions. Subsequently, we employ models like CLIP and BLIP2 to predict specific timestamps for event occurrences. Our evaluation, conducted using the Charades-STA dataset, demonstrates a significant reduction in temporal hallucinations and an improvement in the quality of event-related responses. This research not only provides a new perspective in addressing a critical limitation of MLLMs but also contributes a quantitatively measurable method for evaluating MLLMs in the context of temporal-related questions.