Spatiotemporal Sycophancy: Negation-Based Gaslighting in Video Large Language Models
This work highlights a critical robustness failure in Vid-LLMs for conversational video understanding, relevant to developers and users of interactive AI systems.
The paper identifies and systematically evaluates spatiotemporal sycophancy in Video Large Language Models (Vid-LLMs), where models retract correct visual judgments and fabricate false explanations under negation-based gaslighting. Experiments on the proposed GasVideo-1000 benchmark show this vulnerability is pervasive and severe across state-of-the-art models, with prompt-level grounding providing only partial mitigation.
Video Large Language Models (Vid-LLMs) have demonstrated remarkable performance in video understanding tasks, yet their robustness under conversational interaction remains largely underexplored. In this paper, we identify spatiotemporal sycophancy, a failure mode in which Vid-LLMs retract initially correct, visually grounded judgments and conform to misleading user feedback under negation-based gaslighting. Rather than merely changing their answers, the models often fabricate unsupported temporal or spatial explanations to justify incorrect revisions. To systematically investigate this phenomenon, we propose a negation-based gaslighting evaluation framework and introduce GasVideo-1000, a curated benchmark designed to probe spatiotemporal sycophancy with clear visual grounding and temporal reasoning requirements. We evaluate a broad range of state-of-the-art open-source and proprietary Vid-LLMs across diverse video understanding tasks. Extensive experiments reveal that vulnerability to negation-based gaslighting is pervasive and severe, even among models with strong baseline performance. While prompt-level grounding constraints can partially mitigate this behavior, they do not reliably prevent hallucinated justifications or belief reversal. Our results indicate that current Vid-LLMs lack robust mechanisms for maintaining grounded spatiotemporal beliefs under adversarial conversational feedback.