Customized determination of stop words using Random Matrix Theory approach
This provides a method for creating customized stop word lists in text analysis, but it is incremental as it builds on existing stop word concepts with a new statistical approach.
The paper tackled the problem of identifying stop words in texts by analyzing word distances using Random Matrix Theory and the Brody distribution, finding that words fitting this distribution with a certain threshold can be classified as uninformative stop words, enabling customizable extraction across languages.
The distances between words calculated in word units are studied and compared with the distributions of the Random Matrix Theory (RMT). It is found that the distribution of distance between the same words can be well described by the single-parameter Brody distribution. Using the Brody distribution fit, we found that the distance between given words in a set of texts can show mixed dynamics, coexisting regular and chaotic regimes. It is found that distributions correctly fitted by the Brody distribution with a certain goodness of the fit threshold can be identifid as stop words, usually considered as the uninformative part of the text. By applying various threshold values for the goodness of fit, we can extract uninformative words from the texts under analysis to the desired extent. On this basis we formulate a fully agnostic recipe that can be used in the creation of a customized set of stop words for texts in any language based on words.