Yes but.. Can ChatGPT Identify Entities in Historical Documents?
This work addresses the problem of applying large language models to historical text analysis for researchers and archivists, but it is incremental as it primarily evaluates an existing model on a new domain.
The paper investigates ChatGPT's ability to perform named entity recognition and classification on historical documents in a zero-shot setting, revealing shortcomings such as inconsistency in entity annotation, entity complexity, and code-switching, with performance impacted by limited access to historical archives.
Large language models (LLMs) have been leveraged for several years now, obtaining state-of-the-art performance in recognizing entities from modern documents. For the last few months, the conversational agent ChatGPT has "prompted" a lot of interest in the scientific community and public due to its capacity of generating plausible-sounding answers. In this paper, we explore this ability by probing it in the named entity recognition and classification (NERC) task in primary sources (e.g., historical newspapers and classical commentaries) in a zero-shot manner and by comparing it with state-of-the-art LM-based systems. Our findings indicate several shortcomings in identifying entities in historical text that range from the consistency of entity annotation guidelines, entity complexity, and code-switching, to the specificity of prompting. Moreover, as expected, the inaccessibility of historical archives to the public (and thus on the Internet) also impacts its performance.