Executable Knowledge Graphs for Replicating AI Research
This addresses the problem of insufficient background knowledge and retrieval limitations for AI researchers and developers, offering an incremental improvement over existing methods.
The paper tackles the challenge of replicating AI research by proposing Executable Knowledge Graphs (xKG), which integrate technical insights and code snippets from scientific literature, resulting in a 10.9% performance gain on PaperBench when integrated with LLM agents.
Replicating AI research is a crucial yet challenging task for large language model (LLM) agents. Existing approaches often struggle to generate executable code, primarily due to insufficient background knowledge and the limitations of retrieval-augmented generation (RAG) methods, which fail to capture latent technical details hidden in referenced papers. Furthermore, previous approaches tend to overlook valuable implementation-level code signals and lack structured knowledge representations that support multi-granular retrieval and reuse. To overcome these challenges, we propose Executable Knowledge Graphs (xKG), a modular and pluggable knowledge base that automatically integrates technical insights, code snippets, and domain-specific knowledge extracted from scientific literature. When integrated into three agent frameworks with two different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on PaperBench, demonstrating its effectiveness as a general and extensible solution for automated AI research replication. Code will released at https://github.com/zjunlp/xKG.