LemmaHead: RAG Assisted Proof Generation Using Large Language Models
This addresses the challenge of enhancing proof generation for AI systems, but it appears incremental as it applies an existing RAG method to a specific domain.
The authors tackled the problem of improving mathematical reasoning in large language models by developing LemmaHead, a retrieval-augmented generation system that supplements queries with textbook context, and tested it on automated theorem proving in Lean, but no concrete performance numbers were provided.
Developing the logic necessary to solve mathematical problems or write mathematical proofs is one of the more difficult objectives for large language models (LLMS). Currently, the most popular methods in literature consists of fine-tuning the model on written mathematical content such as academic publications and textbooks, so that the model can learn to emulate the style of mathematical writing. In this project, we explore the effectiveness of using retrieval augmented generation (RAG) to address gaps in the mathematical reasoning of LLMs. We develop LemmaHead, a RAG knowledge base that supplements queries to the model with relevant mathematical context, with particular focus on context from published textbooks. To measure our model's performance in mathematical reasoning, our testing paradigm focuses on the task of automated theorem proving via generating proofs to a given mathematical claim in the Lean formal language.