Path-RAG: Knowledge-Guided Key Region Retrieval for Open-ended Pathology Visual Question Answering
This work addresses the problem of accurate diagnosis and prognosis in cancer treatment by enhancing AI models for pathology image analysis, though it is incremental as it builds on existing methods like LLaVA-Med and HistoCartography.
The paper tackles the challenge of Open-ended Pathology Visual Question Answering (PathVQA-Open) by proposing Path-RAG, a framework that uses HistoCartography to retrieve domain knowledge from pathology images, improving LLaVA-Med accuracy from 38% to 47% on PathVQA-Open and achieving gains of 32.5% and 30.6% on other datasets.
Accurate diagnosis and prognosis assisted by pathology images are essential for cancer treatment selection and planning. Despite the recent trend of adopting deep-learning approaches for analyzing complex pathology images, they fall short as they often overlook the domain-expert understanding of tissue structure and cell composition. In this work, we focus on a challenging Open-ended Pathology VQA (PathVQA-Open) task and propose a novel framework named Path-RAG, which leverages HistoCartography to retrieve relevant domain knowledge from pathology images and significantly improves performance on PathVQA-Open. Admitting the complexity of pathology image analysis, Path-RAG adopts a human-centered AI approach by retrieving domain knowledge using HistoCartography to select the relevant patches from pathology images. Our experiments suggest that domain guidance can significantly boost the accuracy of LLaVA-Med from 38% to 47%, with a notable gain of 28% for H&E-stained pathology images in the PathVQA-Open dataset. For longer-form question and answer pairs, our model consistently achieves significant improvements of 32.5% in ARCH-Open PubMed and 30.6% in ARCH-Open Books on H\&E images. Our code and dataset is available here (https://github.com/embedded-robotics/path-rag).