Assessing LLMs for Serendipity Discovery in Knowledge Graphs: A Case for Drug Repurposing
This work addresses the need for serendipity-aware question answering in scientific domains like drug repurposing, but it is incremental as it focuses on evaluation rather than proposing a new method.
The paper tackles the problem of assessing large language models (LLMs) for discovering serendipitous insights in knowledge graphs, specifically for drug repurposing, and finds that while LLMs excel at retrieval, they struggle to identify genuinely surprising and valuable discoveries.
Large Language Models (LLMs) have greatly advanced knowledge graph question answering (KGQA), yet existing systems are typically optimized for returning highly relevant but predictable answers. A missing yet desired capacity is to exploit LLMs to suggest surprise and novel ("serendipitious") answers. In this paper, we formally define the serendipity-aware KGQA task and propose the SerenQA framework to evaluate LLMs' ability to uncover unexpected insights in scientific KGQA tasks. SerenQA includes a rigorous serendipity metric based on relevance, novelty, and surprise, along with an expert-annotated benchmark derived from the Clinical Knowledge Graph, focused on drug repurposing. Additionally, it features a structured evaluation pipeline encompassing three subtasks: knowledge retrieval, subgraph reasoning, and serendipity exploration. Our experiments reveal that while state-of-the-art LLMs perform well on retrieval, they still struggle to identify genuinely surprising and valuable discoveries, underscoring a significant room for future improvements. Our curated resources and extended version are released at: https://cwru-db-group.github.io/serenQA.