An Abstract Specification of VoxML as an Annotation Language
This work provides a formal specification for annotating language in embodied AI systems, but it is incremental as it builds on existing VoxML concepts.
The paper tackles the problem of mapping natural language to visualizations by specifying VoxML as an abstract annotation language, and it demonstrates this by annotating linguistic data for human-object interactions to support VoxML's modeling purposes.
VoxML is a modeling language used to map natural language expressions into real-time visualizations using commonsense semantic knowledge of objects and events. Its utility has been demonstrated in embodied simulation environments and in agent-object interactions in situated multimodal human-agent collaboration and communication. It introduces the notion of object affordance (both Gibsonian and Telic) from HRI and robotics, as well as the concept of habitat (an object's context of use) for interactions between a rational agent and an object. This paper aims to specify VoxML as an annotation language in general abstract terms. It then shows how it works on annotating linguistic data that express visually perceptible human-object interactions. The annotation structures thus generated will be interpreted against the enriched minimal model created by VoxML as a modeling language while supporting the modeling purposes of VoxML linguistically.