Investigating and Enhancing the Robustness of Large Multimodal Models Against Temporal Inconsistency
This addresses robustness issues in LMMs for video comprehension applications, representing an incremental improvement with a novel method for a known bottleneck.
The paper tackles the problem of temporal analysis robustness in Large Multimodal Models (LMMs) by introducing a benchmark with temporal inconsistency perturbations, finding that 16 mainstream models rely too much on prior knowledge and text, and proposing a method that enhances robustness effectively.
Large Multimodal Models (LMMs) have recently demonstrated impressive performance on general video comprehension benchmarks. Nevertheless, for broader applications, the robustness of their temporal analysis capability needs to be thoroughly investigated yet predominantly ignored. Motivated by this, we propose a novel temporal robustness benchmark (TemRobBench), which introduces temporal inconsistency perturbations separately at the visual and textual modalities to assess the robustness of models. We evaluate 16 mainstream LMMs and find that they exhibit over-reliance on prior knowledge and textual context in adversarial environments, while ignoring the actual temporal dynamics in the video. To mitigate this issue, we design panoramic direct preference optimization (PanoDPO), which encourages LMMs to incorporate both visual and linguistic feature preferences simultaneously. Experimental results show that PanoDPO can effectively enhance the model's robustness and reliability in temporal analysis.