Using off-the-shelf LLMs to query enterprise data by progressively revealing ontologies
This addresses a practical bottleneck for enterprises using LLMs to query data, though it is an incremental improvement over existing zero-shot prompting methods.
The paper tackles the problem of querying enterprise data with large ontologies that exceed LLM token limits by proposing a method to incrementally reveal only necessary parts of the ontology, improving query accuracy without fine-tuning.
Ontologies are known to improve the accuracy of Large Language Models (LLMs) when translating natural language queries into a formal query language like SQL or SPARQL. There are two ways to leverage ontologies when working with LLMs. One is to fine-tune the model, i.e., to enhance it with specific domain knowledge. Another is the zero-shot prompting approach, where the ontology is provided as part of the input question. Unfortunately, modern enterprises typically have ontologies that are too large to fit in a prompt due to LLM's token size limitations. We present a solution that incrementally reveals "just enough" of an ontology that is needed to answer a given question.