What can a cook in Italy teach a mechanic in India? Action Recognition Generalisation Over Scenarios and Locations
This addresses a critical generalization challenge in action recognition for computer vision applications, though it is incremental as it builds on existing domain generalization methods.
The paper tackles the problem of action recognition models failing to generalize to unseen scenarios and locations, and shows that their proposed CIR method outperforms prior domain generalization works on all test splits of the new ARGO1M dataset containing 1.1M video clips.
We propose and address a new generalisation problem: can a model trained for action recognition successfully classify actions when they are performed within a previously unseen scenario and in a previously unseen location? To answer this question, we introduce the Action Recognition Generalisation Over scenarios and locations dataset (ARGO1M), which contains 1.1M video clips from the large-scale Ego4D dataset, across 10 scenarios and 13 locations. We demonstrate recognition models struggle to generalise over 10 proposed test splits, each of an unseen scenario in an unseen location. We thus propose CIR, a method to represent each video as a Cross-Instance Reconstruction of videos from other domains. Reconstructions are paired with text narrations to guide the learning of a domain generalisable representation. We provide extensive analysis and ablations on ARGO1M that show CIR outperforms prior domain generalisation works on all test splits. Code and data: https://chiaraplizz.github.io/what-can-a-cook/.