VLM-KG: Multimodal Radiology Knowledge Graph Generation
This addresses the need for better knowledge graph generation in radiology to enhance downstream tasks, though it is incremental as it builds on existing VLM capabilities.
The paper tackled the problem of generating radiology-specific knowledge graphs, which is challenging due to specialized language and limited data, by proposing a multimodal VLM-based framework that outperforms previous unimodal methods.
Vision-Language Models (VLMs) have demonstrated remarkable success in natural language generation, excelling at instruction following and structured output generation. Knowledge graphs play a crucial role in radiology, serving as valuable sources of factual information and enhancing various downstream tasks. However, generating radiology-specific knowledge graphs presents significant challenges due to the specialized language of radiology reports and the limited availability of domain-specific data. Existing solutions are predominantly unimodal, meaning they generate knowledge graphs only from radiology reports while excluding radiographic images. Additionally, they struggle with long-form radiology data due to limited context length. To address these limitations, we propose a novel multimodal VLM-based framework for knowledge graph generation in radiology. Our approach outperforms previous methods and introduces the first multimodal solution for radiology knowledge graph generation.