RareAct: A video dataset of unusual interactions
This addresses the problem of evaluating action recognition models for unlikely compositions, primarily for researchers in computer vision, but it is incremental as it focuses on dataset creation and benchmarking.
The authors introduced RareAct, a manually annotated video dataset of 122 unusual actions like 'blend phone', to evaluate zero-shot and few-shot compositionality in action recognition models, showing it remains a challenging and unsolved task with benchmarks using a state-of-the-art pretrained model.
This paper introduces a manually annotated video dataset of unusual actions, namely RareAct, including actions such as "blend phone", "cut keyboard" and "microwave shoes". RareAct aims at evaluating the zero-shot and few-shot compositionality of action recognition models for unlikely compositions of common action verbs and object nouns. It contains 122 different actions which were obtained by combining verbs and nouns rarely co-occurring together in the large-scale textual corpus from HowTo100M, but that frequently appear separately. We provide benchmarks using a state-of-the-art HowTo100M pretrained video and text model and show that zero-shot and few-shot compositionality of actions remains a challenging and unsolved task.