Situational Grounding within Multimodal Simulations
This work addresses the challenge of developing AI systems that can reason effectively in complex, real-time simulation environments, which is incremental as it builds on existing simulation technologies.
The paper tackles the problem of embodied spatial reasoning in open-world environments by proposing a formal model of object and event semantics that makes simulation platforms tractable for AI systems, enabling qualitative reasoning, concept learning, and human communication.
In this paper, we argue that simulation platforms enable a novel type of embodied spatial reasoning, one facilitated by a formal model of object and event semantics that renders the continuous quantitative search space of an open-world, real-time environment tractable. We provide examples for how a semantically-informed AI system can exploit the precise, numerical information provided by a game engine to perform qualitative reasoning about objects and events, facilitate learning novel concepts from data, and communicate with a human to improve its models and demonstrate its understanding. We argue that simulation environments, and game engines in particular, bring together many different notions of "simulation" and many different technologies to provide a highly-effective platform for developing both AI systems and tools to experiment in both machine and human intelligence.