SPIKE-RL: Video-LLMs meet Bayesian Surprise
This work addresses a domain-specific problem for video understanding by improving how Video-LLMs process videos, though it is incremental as it builds on existing methods with novel optimizations.
The authors tackled the problem of Video-LLMs missing critical moments in videos by introducing SPIKE and SPIKE-RL, which use Bayesian Surprise to localize surprising events and optimize frame sampling, achieving consistent performance gains on five downstream benchmarks over uniform sampling.
Real-world videos often show routine activities punctuated by memorable, surprising events. However, most Video-LLMs process videos by sampling frames uniformly, likely missing critical moments that define a video's narrative. We introduce SPIKE, an inference-time framework that quantifies Bayesian Surprise as the belief update triggered by new visual evidence in the video stream, identifying moments where new visual evidence conflicts with prior beliefs. SPIKE effectively localizes surprise in videos, strongly correlated with humans on positive (FunQA) and negative (Oops!) surprise benchmarks. Since the beliefs of zero-shot Video-LLMs are often suboptimal, we develop SPIKE-RL, which leverages GRPO to optimize belief hypotheses based on a reward signal from the video caption. SPIKE and SPIKE-RL guide query-agnostic surprise-weighted frame sampling, which allocates more frames to interesting moments in the video. With this strategy, we achieve consistent performance gains on five downstream benchmarks over uniform sampling. By enabling Video-LLMs to track beliefs and register surprise, our work paves the way for more robust models that can revise their understanding in response to new information.