3D Spatial Understanding in MLLMs: Disambiguation and Evaluation
This addresses the challenge of contextual object localization and disambiguation for collaborative robotic systems, representing an incremental improvement over existing methods.
The paper tackles the problem of MLLMs struggling to provide precise instructions for localizing and disambiguating objects in complex 3D environments, proposing techniques that achieve state-of-the-art performance on conventional metrics and improve 3D spatial understanding.
Multimodal Large Language Models (MLLMs) have made significant progress in tasks such as image captioning and question answering. However, while these models can generate realistic captions, they often struggle with providing precise instructions, particularly when it comes to localizing and disambiguating objects in complex 3D environments. This capability is critical as MLLMs become more integrated with collaborative robotic systems. In scenarios where a target object is surrounded by similar objects (distractors), robots must deliver clear, spatially-aware instructions to guide humans effectively. We refer to this challenge as contextual object localization and disambiguation, which imposes stricter constraints than conventional 3D dense captioning, especially regarding ensuring target exclusivity. In response, we propose simple yet effective techniques to enhance the model's ability to localize and disambiguate target objects. Our approach not only achieves state-of-the-art performance on conventional metrics that evaluate sentence similarity, but also demonstrates improved 3D spatial understanding through 3D visual grounding model.