T-RAG: Lessons from the LLM Trenches
This addresses the need for secure, on-premise LLM applications in enterprise settings, though it is incremental as it builds on existing RAG frameworks with domain-specific customizations.
The authors tackled the problem of building robust question-answering systems over private enterprise documents by combining Retrieval-Augmented Generation (RAG) with a finetuned open-source LLM and a tree structure for entity hierarchies, resulting in better performance than simple RAG or finetuning alone as shown in evaluations like the Needle in a Haystack test.
Large Language Models (LLM) have shown remarkable language capabilities fueling attempts to integrate them into applications across a wide range of domains. An important application area is question answering over private enterprise documents where the main considerations are data security, which necessitates applications that can be deployed on-prem, limited computational resources and the need for a robust application that correctly responds to queries. Retrieval-Augmented Generation (RAG) has emerged as the most prominent framework for building LLM-based applications. While building a RAG is relatively straightforward, making it robust and a reliable application requires extensive customization and relatively deep knowledge of the application domain. We share our experiences building and deploying an LLM application for question answering over private organizational documents. Our application combines the use of RAG with a finetuned open-source LLM. Additionally, our system, which we call Tree-RAG (T-RAG), uses a tree structure to represent entity hierarchies within the organization. This is used to generate a textual description to augment the context when responding to user queries pertaining to entities within the organization's hierarchy. Our evaluations, including a Needle in a Haystack test, show that this combination performs better than a simple RAG or finetuning implementation. Finally, we share some lessons learned based on our experiences building an LLM application for real-world use.