Robust Grounding with MLLMs against Occlusion and Small Objects via Language-guided Semantic Cues
For multimodal large language models (MLLMs) used in visual grounding, this work addresses the underexplored challenge of robustness in crowded scenes with occlusion and small objects.
The paper tackles the problem of grounding objects in crowded scenes where occlusion and small objects degrade performance. By introducing Language-Guided Semantic Cues (LGSCs) extracted from text embeddings and reintegrated into the visual pipeline, the method improves grounding accuracy in crowded scenes.
While Multimodal Large Language Models (MLLMs) have enhanced grounding capabilities in general scenes, their robustness in crowded scenes remains underexplored. Crowded scenes entail visual challenges (i.e., occlusion and small objects), which impair object semantics and degrade grounding performance. In contrast, language expressions are immune to such degradation and preserve object semantics. In light of these observations, we propose a novel method that overcomes such constraints by leveraging Language-Guided Semantic Cues (LGSCs). Specifically, our approach introduces a Semantic Cue Extractor (SCE) to derive semantic cues of objects from the visual pipeline of an MLLM. We then guide these cues using corresponding text embeddings to produce LGSCs as linguistic semantic priors. Subsequently, they are reintegrated into the original visual pipeline to refine object semantics. Extensive experiments and analyses demonstrate that incorporating LGSCs into an MLLM effectively improves grounding accuracy in crowded scenes.