DART: Difficulty-Adaptive Routing for Zero-Shot Video Temporal Grounding
For zero-shot video temporal grounding, DART provides a practical method to handle complex multi-stage queries without task-specific training.
DART addresses the reasoning gap in zero-shot video temporal grounding by using difficulty-aware routing: simple queries take a fast path, complex ones a slow path with structured reasoning. It achieves state-of-the-art zero-shot performance on Charades-STA and ActivityNet Captions, improving mIoU by up to 3.5 points while using over 7 times fewer frames.
Zero-shot video temporal grounding (VTG) localizes events in untrimmed videos from natural language queries without task-specific training. Existing methods rely on frame-query feature matching, which suffices for simple events but struggles with complex multi-stage queries that require understanding temporal ordering and causal structure -- a disparity we call the reasoning gap. We propose DART (Difficulty-Adaptive Routing for Temporal Grounding), which bridges this gap by coupling difficulty-aware routing with structured reasoning in large vision-language models. A query-conditioned Determinantal Point Process (DPP) serves a dual role: selecting diverse, query-relevant keyframes as temporal evidence, and providing spectral entropy as a difficulty indicator. Simple queries are routed to a Fast path for direct prediction, while complex queries follow a Slow path with Temporal Markup Prompting, which decomposes localization into global event analysis, per-frame temporal role annotation, and boundary extraction. On Charades-STA and ActivityNet Captions, DART achieves state-of-the-art zero-shot performance across both identically distributed and multiple out-of-distribution settings, improving mIoU by up to 3.5 points over the strongest baseline while using over 7 times fewer frames. The project homepage is available at https://dart-vtg.github.io/.