PathVG: A New Benchmark and Dataset for Pathology Visual Grounding
This work addresses the problem of region-level detection in computational pathology for AI-assisted diagnostics, but it is incremental as it builds on existing visual grounding methods with domain-specific adaptations.
The authors introduced PathVG, a new benchmark for pathology visual grounding, and created the RefPath dataset with 27,610 images and 33,500 language-grounded boxes to address the lack of region-level detection in computational pathology. Their proposed PKNet model, which uses LLMs to convert implicit pathological terms into explicit features, achieved state-of-the-art performance on this benchmark.
With the rapid development of computational pathology, many AI-assisted diagnostic tasks have emerged. Cellular nuclei segmentation can segment various types of cells for downstream analysis, but it relies on predefined categories and lacks flexibility. Moreover, pathology visual question answering can perform image-level understanding but lacks region-level detection capability. To address this, we propose a new benchmark called Pathology Visual Grounding (PathVG), which aims to detect regions based on expressions with different attributes. To evaluate PathVG, we create a new dataset named RefPath which contains 27,610 images with 33,500 language-grounded boxes. Compared to visual grounding in other domains, PathVG presents pathological images at multi-scale and contains expressions with pathological knowledge. In the experimental study, we found that the biggest challenge was the implicit information underlying the pathological expressions. Based on this, we proposed Pathology Knowledge-enhanced Network (PKNet) as the baseline model for PathVG. PKNet leverages the knowledge-enhancement capabilities of Large Language Models (LLMs) to convert pathological terms with implicit information into explicit visual features, and fuses knowledge features with expression features through the designed Knowledge Fusion Module (KFM). The proposed method achieves state-of-the-art performance on the PathVG benchmark.