Explaining Vision and Language through Graphs of Events in Space and Time
This addresses the problem of bridging vision and language for AI researchers, though it appears incremental as it complements existing deep learning models.
The paper tackles the lack of a common explainable representation between vision and language by proposing the Graph of Events in Space and Time (GEST), which improves video generation from text and semantic text comparisons.
Artificial Intelligence makes great advances today and starts to bridge the gap between vision and language. However, we are still far from understanding, explaining and controlling explicitly the visual content from a linguistic perspective, because we still lack a common explainable representation between the two domains. In this work we come to address this limitation and propose the Graph of Events in Space and Time (GEST), by which we can represent, create and explain, both visual and linguistic stories. We provide a theoretical justification of our model and an experimental validation, which proves that GEST can bring a solid complementary value along powerful deep learning models. In particular, GEST can help improve at the content-level the generation of videos from text, by being easily incorporated into our novel video generation engine. Additionally, by using efficient graph matching techniques, the GEST graphs can also improve the comparisons between texts at the semantic level.