Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction
For researchers in multimodal AI and GUI agents, this work provides a new benchmark and a keyframe extraction method that bridges video understanding and downstream agentic tasks, though the improvements are incremental.
The paper introduces VG-GUIBench, a benchmark to evaluate MLLM-based GUI agents on video-guided agentic tasks, and proposes TASKER, a keyframe extraction algorithm that improves performance on both VideoQA and agentic tasks, outperforming baselines by 2.0% on EgoSchema and 1.8% on NExT-QA.
Video understanding is a fundamental capability for multimodal intelligence, and recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance on Video Question Answering (VideoQA) benchmarks. However, existing benchmarks primarily evaluate whether models can perceive shallow visual cues, while rarely examining whether MLLMs can learn deeper knowledge or procedural skills from video tutorials and generalize them to downstream long-horizon agentic tasks. To address this gap, we introduce VG-GUIBench (Video-Guided GUI Benchmark), a new benchmark designed to evaluate whether MLLM-based GUI agents can follow video tutorials to complete corresponding GUI interactive tasks. Furthermore, we observe that the performance of models on both VideoQA and video-guided agentic tasks critically depends on effective keyframe extraction. Based on this observation, we propose TASKER (Task-driven And Scene-aware Keyframe searchER), a keyframe extraction algorithm that jointly considers task relevance and scene dynamics to identify informative frames. Experimental results demonstrate that TASKER achieves significant performance improvements on both VideoQA and video-guided agentic task benchmarks, outperforming the best baseline by 2.0% on the EgoSchema fullset and 1.8% on the NExT-QA dataset, respectively. These results further highlight the potential of generalized keyframe extraction methods for video understanding tasks. Our code and data are available at https://github.com/VG-GUI-TASKER/VG-GUI-TASKER.