Supporting Mobile Multimodal Interaction with a Rule-Based Framework
This addresses the problem of high development effort for multimodal interaction in mobile devices, primarily for researchers and developers, though it appears incremental as it builds on existing technical capabilities.
The paper tackles the challenge of building multimodal applications on mobile platforms by introducing a rule-based framework that simplifies and accelerates their creation for prototyping and research, validated through four demonstrations of varying complexity.
Multimodality can make (especially mobile) device interaction more efficient. Sensors and communication capabilities of modern smartphones and tablets lay the technical basis for its implementation. Still, mobile platforms do not make multimodal interaction support trivial. Building multimodal applications requires various APIs with different paradigms, high-level interpretation of contextual data, and a method for fusing individual inputs and outputs. To reduce this effort, we created a framework that simplifies and accelerates the creation of multimodal applications for prototyping and research. It provides an abstraction of information representations in different modalities, unifies access to implicit and explicit information, and wires together the logic behind context-sensitive modality switches. In the paper, we present the structure and features of our framework, and validate it by four implemented demonstrations of different complexity.