The Anatomy of a Search and Mining System for Digital Archives
This system assists historians and linguists in analyzing digital archives, but it is incremental as it adapts existing methods to a specific domain.
The authors tackled the problem of quantifying textual corpora for digital humanities research by developing Samtla, a system that uses a character-based n-gram language model for flexible, language-agnostic query processing, with evaluation showing improved ranking performance through crowdsourcing.
Samtla (Search And Mining Tools with Linguistic Analysis) is a digital humanities system designed in collaboration with historians and linguists to assist them with their research work in quantifying the content of any textual corpora through approximate phrase search and document comparison. The retrieval engine uses a character-based n-gram language model rather than the conventional word-based one so as to achieve great flexibility in language agnostic query processing. The index is implemented as a space-optimised character-based suffix tree with an accompanying database of document content and metadata. A number of text mining tools are integrated into the system to allow researchers to discover textual patterns, perform comparative analysis, and find out what is currently popular in the research community. Herein we describe the system architecture, user interface, models and algorithms, and data storage of the Samtla system. We also present several case studies of its usage in practice together with an evaluation of the systems' ranking performance through crowdsourcing.