Pairing Analogy-Augmented Generation with Procedural Memory for Procedural Q&A
This addresses the challenge of procedural knowledge synthesis for AI systems, offering a domain-specific improvement in task planning and Q&A.
The paper tackles the problem of large language models struggling to synthesize information for complex procedural tasks by introducing a novel formalism and dataset (LCStep) and proposing analogy-augmented generation (AAG) with a procedural memory store. The result shows that AAG outperforms few-shot and RAG baselines on multiple datasets, as validated by LLM-based and human evaluations.
Large language models struggle to synthesize disparate pieces of information into a coherent plan when approaching a complex procedural task. In this work, we introduce a novel formalism and structure for such procedural knowledge. Based on this formalism, we present a novel procedural knowledge dataset called LCStep, which we created from LangChain tutorials. To leverage this procedural knowledge to solve new tasks, we propose analogy-augmented generation (AAG), which draws inspiration from the human ability to assimilate past experiences to solve unfamiliar problems. AAG uses a custom procedure memory store to retrieve and adapt specialized domain knowledge to answer new procedural tasks. We demonstrate that AAG outperforms few-shot and RAG baselines on LCStep, RecipeNLG, and CHAMP datasets under a pairwise LLM-based evaluation, corroborated by human evaluation in the case of RecipeNLG.