Disentangled Action Recognition with Knowledge Bases
This work addresses the scalability and generalization challenges in video action recognition for applications like surveillance and robotics, though it is incremental as it builds on prior knowledge graph methods.
The paper tackles the problem of compositional action recognition in videos, where models must generalize to novel verb-noun combinations unseen during training, by proposing DARK, a factorized model that uses knowledge graphs to disentangle verb and noun features and apply type constraints, achieving state-of-the-art performance on the Charades dataset and introducing a new, larger benchmark on Epic-kitchen.
Action in video usually involves the interaction of human with objects. Action labels are typically composed of various combinations of verbs and nouns, but we may not have training data for all possible combinations. In this paper, we aim to improve the generalization ability of the compositional action recognition model to novel verbs or novel nouns that are unseen during training time, by leveraging the power of knowledge graphs. Previous work utilizes verb-noun compositional action nodes in the knowledge graph, making it inefficient to scale since the number of compositional action nodes grows quadratically with respect to the number of verbs and nouns. To address this issue, we propose our approach: Disentangled Action Recognition with Knowledge-bases (DARK), which leverages the inherent compositionality of actions. DARK trains a factorized model by first extracting disentangled feature representations for verbs and nouns, and then predicting classification weights using relations in external knowledge graphs. The type constraint between verb and noun is extracted from external knowledge bases and finally applied when composing actions. DARK has better scalability in the number of objects and verbs, and achieves state-of-the-art performance on the Charades dataset. We further propose a new benchmark split based on the Epic-kitchen dataset which is an order of magnitude bigger in the numbers of classes and samples, and benchmark various models on this benchmark.