Router-Suggest: Dynamic Routing for Multimodal Auto-Completion in Visually-Grounded Dialogs
This addresses the need for efficient and context-aware auto-completion in applications like digital assistants and chatbots, though it is incremental as it builds on existing models with a routing mechanism.
The paper tackled the problem of multimodal auto-completion in visually-grounded dialogs by introducing a router framework that dynamically selects between textual and vision-language models, achieving a 2.3x to 10x speedup over the best-performing VLM and improving user satisfaction in typing effort and completion quality.
Real-time multimodal auto-completion is essential for digital assistants, chatbots, design tools, and healthcare consultations, where user inputs rely on shared visual context. We introduce Multimodal Auto-Completion (MAC), a task that predicts upcoming characters in live chats using partially typed text and visual cues. Unlike traditional text-only auto-completion (TAC), MAC grounds predictions in multimodal context to better capture user intent. To enable this task, we adapt MMDialog and ImageChat to create benchmark datasets. We evaluate leading vision-language models (VLMs) against strong textual baselines, highlighting trade-offs in accuracy and efficiency. We present Router-Suggest, a router framework that dynamically selects between textual models and VLMs based on dialog context, along with a lightweight variant for resource-constrained environments. Router-Suggest achieves a 2.3x to 10x speedup over the best-performing VLM. A user study shows that VLMs significantly excel over textual models on user satisfaction, notably saving user typing effort and improving the quality of completions in multi-turn conversations. These findings underscore the need for multimodal context in auto-completions, leading to smarter, user-aware assistants.