Talking about the Moving Image: A Declarative Model for Image Schema Based Embodied Perception Grounding and Language Generation
This work addresses the need for human-interactive and evidence-based qualitative analysis in domains like film and smart environments, though it appears incremental as it builds on existing theories of image schemas and declarative models.
The paper tackles the problem of grounding dynamic visual imagery in embodied perception and generating analytical natural language summaries by introducing a declarative model based on image schemas and spatio-linguistic abstractions, implemented in Constraint Logic Programming to enable inference and querying with deep semantics.
We present a general theory and corresponding declarative model for the embodied grounding and natural language based analytical summarisation of dynamic visuo-spatial imagery. The declarative model ---ecompassing spatio-linguistic abstractions, image schemas, and a spatio-temporal feature based language generator--- is modularly implemented within Constraint Logic Programming (CLP). The implemented model is such that primitives of the theory, e.g., pertaining to space and motion, image schemata, are available as first-class objects with `deep semantics' suited for inference and query. We demonstrate the model with select examples broadly motivated by areas such as film, design, geography, smart environments where analytical natural language based externalisations of the moving image are central from the viewpoint of human interaction, evidence-based qualitative analysis, and sensemaking. Keywords: moving image, visual semantics and embodiment, visuo-spatial cognition and computation, cognitive vision, computational models of narrative, declarative spatial reasoning