Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QA
This work addresses the problem of inaccurate AI responses in knowledge-intensive vertical domains like EDA for engineers and developers, though it is incremental as it adapts existing RAG methods to a specific domain.
The paper tackles the challenge of applying retrieval augmented generation (RAG) to electronic design automation (EDA) tool documentation by proposing a customized RAG framework with domain-specific techniques, achieving superior performance on a new benchmark and a commercial tool compared to state-of-the-art methods.
Retrieval augmented generation (RAG) enhances the accuracy and reliability of generative AI models by sourcing factual information from external databases, which is extensively employed in document-grounded question-answering (QA) tasks. Off-the-shelf RAG flows are well pretrained on general-purpose documents, yet they encounter significant challenges when being applied to knowledge-intensive vertical domains, such as electronic design automation (EDA). This paper addresses such issue by proposing a customized RAG framework along with three domain-specific techniques for EDA tool documentation QA, including a contrastive learning scheme for text embedding model fine-tuning, a reranker distilled from proprietary LLM, and a generative LLM fine-tuned with high-quality domain corpus. Furthermore, we have developed and released a documentation QA evaluation benchmark, ORD-QA, for OpenROAD, an advanced RTL-to-GDSII design platform. Experimental results demonstrate that our proposed RAG flow and techniques have achieved superior performance on ORD-QA as well as on a commercial tool, compared with state-of-the-arts. The ORD-QA benchmark and the training dataset for our customized RAG flow are open-source at https://github.com/lesliepy99/RAG-EDA.