LangHOPS: Language Grounded Hierarchical Open-Vocabulary Part Segmentation
This addresses the problem of segmenting hierarchical object parts from open-vocabulary categories for computer vision applications, representing a novel method rather than an incremental improvement.
The paper tackles open-vocabulary object-part instance segmentation by proposing LangHOPS, a framework that uses a Multimodal Large Language Model to ground object-part hierarchies in language space, achieving state-of-the-art results with improvements of 5.5% AP in-domain and 4.8% AP cross-dataset on PartImageNet, and 2.5% mIOU on unseen parts in ADE20K.
We propose LangHOPS, the first Multimodal Large Language Model (MLLM) based framework for open-vocabulary object-part instance segmentation. Given an image, LangHOPS can jointly detect and segment hierarchical object and part instances from open-vocabulary candidate categories. Unlike prior approaches that rely on heuristic or learnable visual grouping, our approach grounds object-part hierarchies in language space. It integrates the MLLM into the object-part parsing pipeline to leverage its rich knowledge and reasoning capabilities, and link multi-granularity concepts within the hierarchies. We evaluate LangHOPS across multiple challenging scenarios, including in-domain and cross-dataset object-part instance segmentation, and zero-shot semantic segmentation. LangHOPS achieves state-of-the-art results, surpassing previous methods by 5.5% Average Precision (AP) (in-domain) and 4.8% (cross-dataset) on the PartImageNet dataset and by 2.5% mIOU on unseen object parts in ADE20K (zero-shot). Ablation studies further validate the effectiveness of the language-grounded hierarchy and MLLM driven part query refinement strategy. The code will be released here.