Perception Test 2023: A Summary of the First Challenge And Outcome
This work provides a comprehensive benchmark for video models, addressing the need for standardized evaluation in multimodal AI research, though it is incremental as it reports on an existing challenge.
The paper summarizes the first Perception Test challenge, which benchmarked state-of-the-art video models across six multimodal tasks, including object tracking and video question-answering, to evaluate performance on the Perception Test benchmark.
The First Perception Test challenge was held as a half-day workshop alongside the IEEE/CVF International Conference on Computer Vision (ICCV) 2023, with the goal of benchmarking state-of-the-art video models on the recently proposed Perception Test benchmark. The challenge had six tracks covering low-level and high-level tasks, with both a language and non-language interface, across video, audio, and text modalities, and covering: object tracking, point tracking, temporal action localisation, temporal sound localisation, multiple-choice video question-answering, and grounded video question-answering. We summarise in this report the task descriptions, metrics, baselines, and results.