Lumos : Empowering Multimodal LLMs with Scene Text Recognition
This addresses the challenge of text understanding in multimodal AI for applications like augmented reality or robotics, though it appears incremental as it combines existing STR and MM-LLM components.
The paper tackles the problem of enabling multimodal question-answering systems to understand text in first-person images by introducing Lumos, which integrates Scene Text Recognition (STR) to extract and augment input for a Multimodal Large Language Model (MM-LLM), resulting in high quality and efficiency as demonstrated in evaluations.
We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency.