Scaling Expert Feedback with Reflective Edit Propagation in Compositional Knowledge Bases
For organizations curating domain-specific knowledge bases, RAID addresses the bottleneck of scaling expert validation by automating propagation of edits, though the results are preliminary with no concrete performance numbers.
RAID transforms individual expert edits into systematic knowledge updates by inferring the semantic intent behind a single edit and propagating corrections across a compositional knowledge base, demonstrating technical feasibility in capturing expert intent and potential to scale specialized expertise.
Domain-specific knowledge bases (KBs) encode vertical expertise and proprietary information that organizations depend on, but curating them at scale is a persistent challenge. Although Large Language Models (LLMs) can draft initial entries efficiently, technical accuracy still requires human expert validation, and reviewing entries one by one at scale is impractical. We present Reflective Agent for Identifier Dictionary (RAID), a novel system that transforms individual expert edits into systematic knowledge updates. Unlike traditional "correct-and-save" paradigms, RAID utilizes a reflective agent to infer the underlying semantic intent behind a single expert edit and propagates that correction across the entire KB through a three-step architecture: Intent Inference, Reflection-based Planning, and User Controlled Execution. We evaluated the reflection and propagation performance on a public dataset and conducted a user study with subject matter experts with proprietary data. The evaluation shows RAID's technical feasibility in capturing expert intent and its potential to scale specialized expertise across industrial knowledge bases.