OnlineSI: Taming Large Language Model for Online 3D Understanding and Grounding
This work addresses the challenge of real-time spatial reasoning for embodied AI systems, representing an incremental advancement in deploying such models in dynamic environments.
The paper tackles the problem of enabling multimodal large language models to continuously understand and ground 3D spatial information from video streams, achieving effective results as demonstrated on two datasets with a new Fuzzy F1-Score metric.
In recent years, researchers have increasingly been interested in how to enable Multimodal Large Language Models (MLLM) to possess spatial understanding and reasoning capabilities. However, most existing methods overlook the importance of the ability to continuously work in an ever-changing world, and lack the possibility of deployment on embodied systems in real-world environments. In this work, we introduce OnlineSI, a framework that can continuously improve its spatial understanding of its surroundings given a video stream. Our core idea is to maintain a finite spatial memory to retain past observations, ensuring the computation required for each inference does not increase as the input accumulates. We further integrate 3D point cloud information with semantic information, helping MLLM to better locate and identify objects in the scene. To evaluate our method, we introduce the Fuzzy $F_1$-Score to mitigate ambiguity, and test our method on two representative datasets. Experiments demonstrate the effectiveness of our method, paving the way towards real-world embodied systems.