Detection-Fusion for Knowledge Graph Extraction from Videos
This addresses the challenge of video understanding for applications requiring structured, computer-processable annotations, though it appears incremental by building on existing detection and fusion techniques.
The paper tackles the problem of extracting semantic content from videos by proposing a method to annotate videos with knowledge graphs, avoiding the shortcomings of natural language descriptions. The result is a deep-learning-based model that predicts pairs of individuals and their relations, with an extension for incorporating background knowledge.
One of the challenging tasks in the field of video understanding is extracting semantic content from video inputs. Most existing systems use language models to describe videos in natural language sentences, but this has several major shortcomings. Such systems can rely too heavily on the language model component and base their output on statistical regularities in natural language text rather than on the visual contents of the video. Additionally, natural language annotations cannot be readily processed by a computer, are difficult to evaluate with performance metrics and cannot be easily translated into a different natural language. In this paper, we propose a method to annotate videos with knowledge graphs, and so avoid these problems. Specifically, we propose a deep-learning-based model for this task that first predicts pairs of individuals and then the relations between them. Additionally, we propose an extension of our model for the inclusion of background knowledge in the construction of knowledge graphs.