Corpus analysis without prior linguistic knowledge - unsupervised mining of phrases and subphrase structure
This work addresses the challenge of language-independent text analysis for applications like text mining and search engines, though it is incremental in advancing unsupervised techniques.
The paper tackles the problem of identifying phrases and subphrase structures in natural language without prior linguistic knowledge, using unsupervised methods to automatically mine these elements from corpora, with results demonstrated across multiple languages.
When looking at the structure of natural language, "phrases" and "words" are central notions. We consider the problem of identifying such "meaningful subparts" of language of any length and underlying composition principles in a completely corpus-based and language-independent way without using any kind of prior linguistic knowledge. Unsupervised methods for identifying "phrases", mining subphrase structure and finding words in a fully automated way are described. This can be considered as a step towards automatically computing a "general dictionary and grammar of the corpus". We hope that in the long run variants of our approach turn out to be useful for other kind of sequence data as well, such as, e.g., speech, genom sequences, or music annotation. Even if we are not primarily interested in immediate applications, results obtained for a variety of languages show that our methods are interesting for many practical tasks in text mining, terminology extraction and lexicography, search engine technology, and related fields.