If the Sources Could Talk: Evaluating Large Language Models for Research Assistance in History
This provides a conversational tool for historians and humanities researchers to interact with textual archives, though it is incremental as it builds on existing LLM capabilities with domain-specific augmentations.
The paper tackles the problem of assisting historians and humanities researchers by evaluating large language models (LLMs) augmented with vector embeddings from specialized academic sources, demonstrating their ability to perform question-answering and data extraction tasks on customized corpora not in the training data.
The recent advent of powerful Large-Language Models (LLM) provides a new conversational form of inquiry into historical memory (or, training data, in this case). We show that by augmenting such LLMs with vector embeddings from highly specialized academic sources, a conversational methodology can be made accessible to historians and other researchers in the Humanities. Concretely, we evaluate and demonstrate how LLMs have the ability of assisting researchers while they examine a customized corpora of different types of documents, including, but not exclusive to: (1). primary sources, (2). secondary sources written by experts, and (3). the combination of these two. Compared to established search interfaces for digital catalogues, such as metadata and full-text search, we evaluate the richer conversational style of LLMs on the performance of two main types of tasks: (1). question-answering, and (2). extraction and organization of data. We demonstrate that LLMs semantic retrieval and reasoning abilities on problem-specific tasks can be applied to large textual archives that have not been part of the its training data. Therefore, LLMs can be augmented with sources relevant to specific research projects, and can be queried privately by researchers.