A fully data-driven method to identify (correlated) changes in diachronic corpora
This incremental work addresses the problem of analyzing language change over time for linguists and NLP researchers, potentially improving diachronic POS tagging and complementing existing NLP approaches.
The paper tackles the problem of identifying trends and correlated changes in diachronic text data by adapting and extending an existing method for synchronic corpus similarity, using datasets like the Corpus of Historical American English and Google Ngram Corpora. The result is a computationally cheap, data-driven method that extracts words with pronounced frequency changes, enabling plausible interpretations and linking to historical events.
In this paper, a method for measuring synchronic corpus (dis-)similarity put forward by Kilgarriff (2001) is adapted and extended to identify trends and correlated changes in diachronic text data, using the Corpus of Historical American English (Davies 2010a) and the Google Ngram Corpora (Michel et al. 2010a). This paper shows that this fully data-driven method, which extracts word types that have undergone the most pronounced change in frequency in a given period of time, is computationally very cheap and that it allows interpretations of diachronic trends that are both intuitively plausible and motivated from the perspective of information theory. Furthermore, it demonstrates that the method is able to identify correlated linguistic changes and diachronic shifts that can be linked to historical events. Finally, it can help to improve diachronic POS tagging and complement existing NLP approaches. This indicates that the approach can facilitate an improved understanding of diachronic processes in language change.