A Solution to CVPR'2023 AQTC Challenge: Video Alignment for Multi-Step Inference
This work addresses the challenge of AI assistants providing accurate multi-step instructions from videos, but it is incremental as it builds on existing methods like VideoCLIP and GRU for a specific benchmark.
The paper tackles the problem of video alignment for multi-step inference in the AQTC challenge, achieving second place in the CVPR'2023 competition by improving step-by-step guidance accuracy through enhanced feature reweighting and GRU-based inference.
Affordance-centric Question-driven Task Completion (AQTC) for Egocentric Assistant introduces a groundbreaking scenario. In this scenario, through learning instructional videos, AI assistants provide users with step-by-step guidance on operating devices. In this paper, we present a solution for enhancing video alignment to improve multi-step inference. Specifically, we first utilize VideoCLIP to generate video-script alignment features. Afterwards, we ground the question-relevant content in instructional videos. Then, we reweight the multimodal context to emphasize prominent features. Finally, we adopt GRU to conduct multi-step inference. Through comprehensive experiments, we demonstrate the effectiveness and superiority of our method, which secured the 2nd place in CVPR'2023 AQTC challenge. Our code is available at https://github.com/zcfinal/LOVEU-CVPR23-AQTC.